[R] [Fwd: Re: Coefficients of Logistic Regression from bootstrap - how to get them?]

From: Michal Figurski <figurski_at_mail.med.upenn.edu>
Date: Thu, 24 Jul 2008 10:55:11 -0400


Thank you Frank and all for your advices.

Here I attach the raw data from the Pawinski's paper. I have obtained permission from the corresponding Author to post it here for everyone. The only condition of use is that the Authors retain ownership of the data, and any publication resulting from these data must be managed by them.

The dataset is composed as follows: patient number / MMF dose in [g] / Day of study (since start of drug administration) / MPA concentrations [mg/L] in plasma at following time points: 0, 0.5 ... 12 hours / and the value of AUC(0-12h) calculated using all time-points.

The goal of the analysis, as you can read from the paper, was to estimate the value of AUC using maximum 3 time-points within 2 hours post dose, that is using only 3 of the 4 time-points: 0, 0.5, 1, 2 - but always include the "0" time-point.

In my analysis of similar problem I was also concerned about the fact that data come from several visits of a single patient. I have examined the effect of "PT" with repeated "day" using mixed effects model, and these effects turned out to be insignificant. Do you guys think it is enough justification to use the dataset as if coming from 50 separate patients?

Also, as to estimation of the bias, variance, etc, Pawinski used CI and Sy/x. In my analysis I additionally used RMSE values. Please excuse another naive question, but: do you think it is sufficient information to compare between models and account for bias?

Regarding the "multiple stepwise regression" - according to the cited SPSS manual, there are 5 options to select from. I don't think they used 'stepwise selection' option, because their models were already pre-defined. Variables were pre-selected based on knowledge of pharmacokinetics of this drug and other factors. I think this part I understand pretty well.

I see the Frank's point about recalibration on Fig.2 - although the expectation was set that the prediction be within 15% of the original value. In my opinion it is *very strict* - I actually used 20% in my work. This is because of very high variability and imprecision in the results themselves. These are real biological data and you have to account for errors like analytical errors (HPLC method), timing errors and so on, when you look at these data. In other words, if you take two blood samples at each time-point from a particular patient, and run them, you will 100% certainly get two distinct (although similar) profiles. You will get even more difference, if you run one set of samples on one day, and another set on second day.

Therefore the value of AUC(0-12) itself, to which we compare the predicted AUC, is not 'holy' - some variability here is inherent.

Nevertheless, I see that the Fig.2 may be incorrect, if we look from orthodox statistical perspective. I used the same plots in my work as well - it's too late now. How should I properly estimate the Rsq then?

I greatly appreciate your time and advices in this matter.

--
Michal J. Figurski

Frank E Harrell Jr wrote:
> Gustaf Rydevik wrote:

>> On Wed, Jul 23, 2008 at 4:08 PM, Michal Figurski
>> <figurski_at_mail.med.upenn.edu> wrote:
>>
>> Hi,
>>
>> I believe that you misunderstand the passage. Do you know what
>> multiple stepwise regression is?
>>
>> Since they used SPSS, I copied from
>> http://www.visualstatistics.net/SPSS%20workbook/stepwise_multiple_regression.htm
>>
>>
>> "Stepwise selection is a combination of forward and backward procedures.
>> Step 1
>>
>> The first predictor variable is selected in the same way as in forward
>> selection. If the probability associated with the test of significance
>> is less than or equal to the default .05, the predictor variable with
>> the largest correlation with the criterion variable enters the
>> equation first.
>>
>>
>> Step 2
>>
>> The second variable is selected based on the highest partial
>> correlation. If it can pass the entry requirement (PIN=.05), it also
>> enters the equation.
>>
>> Step 3
>>
>>> From this point, stepwise selection differs from forward selection:
>> the variables already in the equation are examined for removal
>> according to the removal criterion (POUT=.10) as in backward
>> elimination.
>>
>> Step 4
>>
>> Variables not in the equation are examined for entry. Variable
>> selection ends when no more variables meet entry and removal criteria.
>> -----------
>>
>>
>> It is the outcome of this *entire process*,step1-4, that they compare
>> with the outcome of their *entire bootstrap/crossvalidation/selection
>> process*, Step1-4 in the methods section, and find that their approach
>> gives better result
>> What you are doing is only step4 in the article's method
>> section,estimating the parameters of a model *when you already know
>> which variables to include*.It is the way this step is conducted that
>> I am sceptical about.
>>
>> Regards,
>>
>> Gustaf
>>
> > Perfectly stated Gustaf. This is a great example of needing to truly > understand a method to be able to use it in the right context. > > After having read most of the paper by Pawinski et al now, there are > other problems. > > 1. The paper nowhere uses bootstrapping. It uses repeated 2-fold > cross-validation, a procedure not usually recommended. > > 2. The resampling procedure used in the paper treated the 50 > pharmacokinetic profiles on 21 renal transplant patients as if these > were from 50 patients. The cluster bootstrap should have been used > instead. > > 3. Figure 2 showed the fitted regression line to the predicted vs. > observed AUCs. It should have shown the line of identify instead. In > other words, the authors allowed a subtle recalibration to creep into > the analysis (and inverted the x- and y-variables in the plots). The > fitted lines are far enough away from the line of identity as to show > that the predicted values are not well calibrated. The r^2 values > claimed by the authors used the wrong formulas which allowed an > automatic after-the-fact recalibration (new overall slope and intercept > are estimated in the test dataset). Hence the achieved r^2 are misleading.

"PT","Dose","Day",0,0.5,1,2,3,4,6,8,9,10,11,12,"AUC tr" 1,1,2,1.18,2.14,1.42,3.99,3.42,2.09,0.56,4.41,5.32,1.52,0.98,1.18,29.12 1,1,90,2.4,6.4,7.63,5.03,4.56,3.06,1.62,1.09,0.97,1.77,2.72,2.66,35.37 1,1,14,1.23,1.41,1.47,4.29,3.74,2.95,1.27,1.17,1.43,1.91,1.84,2.71,25.40 2,2,2,0.39,1.15,3.3,3.53,4.55,5.19,2.15,2.83,1.2,5.41,6.63,3.09,42.34 2,2,90,3.07,5.71,23.16,5.94,2.49,1.32,2.34,2.96,1.6,4.03,2.74,1.49,49.64 2,2,14,0.58,4.2,13.26,6.7,2.17,0.78,0.97,0.92,0.87,1.48,3.4,4.71,33.66 3,1,2,0.36,0.84,1.49,1.23,1.7,1.15,0.62,0.88,2.82,0.76,0.49,0.36,13.09 3,1,90,0.98,6.41,5.31,2.14,1.47,1.47,1.14,0.66,0.79,1.04,1.35,1.21,20.30 3,1,14,0.43,3.13,3.55,2.02,1.78,1.65,0.97,3.1,2.14,1.26,0.78,0.65,21.71 4,2,2,0.73,1.61,2,2.75,2.76,1.77,1.05,2.16,2.32,2.59,2.76,0.85,24.09 4,2,90,3.63,12.23,5.53,3.09,2.01,2.25,5.34,6.92,7.17,5.36,3.45,3.9,58.64 4,2,14,2.29,3.06,5.25,3.23,2.41,2.65,1.58,1.09,1.51,1.92,1.86,2.96,27.22 5,2,2,1.73,6.74,7.59,5.43,3.74,5.47,6.6,5.85,3.91,2.54,1.97,5.14,59.84 5,2,90,4.13,4.67,10.95,16.94,7.36,4.39,3.56,1.89,2.99,3.77,4.64,13.1,70.37 5,2,14,3.41,4.11,4.18,4.07,4.26,2.76,4.29,3.37,2.66,1.86,2.23,2.47,40.13 6,1,2,1.22,3.44,11.57,1.44,0.82,0.56,1.66,1.93,0.67,0.47,1,1.22,22.77 6,1,90,0.7,0.64,2.29,6.5,1.84,0.74,1.99,1.44,1.39,0.61,0.91,0.72,21.07 6,1,14,0.22,22.63,4.73,0.88,0.46,0.33,0.3,0.33,0.71,1.43,1.35,4.1,23.39 7,2,2,0.72,0.74,0.84,1.6,1.5,1.07,0.71,0.39,0.28,0.57,0.59,0.72,9.69 7,2,90,2.94,6.07,12.18,11.19,4.77,2.76,2.56,2.24,1.7,1.2,0.96,1.25,45.97 7,2,14,3.67,1.43,6.76,4.32,1.85,1.38,1.15,0.69,0.49,1.24,1.34,4.8,23.75 8,1,90,1.98,4.18,5.4,4.22,1.49,1.34,1.82,2.45,1.55,1.23,1.65,2.55,27.38 8,1,14,0.28,1.18,9.03,0.94,0.66,0.43,0.95,0.46,0.33,0.3,0.37,0.21,13.37 9,2,90,5.2,16.98,10.68,3.18,3.18,3.06,4.23,4,2.85,2.37,4.86,6.64,56.61 9,2,14,1.22,3.91,11.38,6.56,3.17,1.89,1.29,2.56,3.1,3.28,4.09,2.86,41.68 10,2,90,0.76,31.22,5.51,2.19,1.33,1.06,0.79,0.59,1.26,3.54,1.99,1.09,34.84 10,2,14,1.94,14.62,7.51,2.7,1.9,4.87,2.3,0.83,0.63,1.34,1.56,1.01,35.21 11,2,90,1.9,20.62,14.48,6.2,3.24,2.58,3.96,4.82,3.86,3.9,4.08,2.91,63.40 11,2,2,0.8,11.69,8.74,3.56,2.62,0.98,1.14,1.45,1.02,0.79,0.7,0.8,27.62 11,2,14,3.14,13.43,14.42,4.5,2.71,5.1,4.19,2.66,1.99,1.48,2.34,3.1,52.91 12,1,90,2.5,2.02,1.65,2.1,2.76,3.44,1.3,1.43,1.11,1.14,1.31,1.56,21.98 12,1,14,2.72,22.68,6.69,5.12,2.5,2.42,0.87,0.68,2.74,10.53,7.98,4.68,54.64 13,2,90,2.75,2.06,2.12,2.32,3,5.25,2.45,2.23,3.19,3.17,3.73,3.08,36.38 13,2,14,1.86,2,5.29,3.12,3.17,2.97,0.84,1,0.36,0.28,0.21,0.23,20.32 14,1,90,1.73,2.05,1.6,6.19,6.11,4.04,1.27,1.31,0.91,0.97,0.8,0.76,28.58 14,1,14,1.45,3.86,3.61,3.3,2.04,1.72,1.27,0.94,0.9,0.83,0.72,0.5,19.57 15,1,90,3.05,2.74,6.99,4.21,3.3,2.45,1.57,3.24,3.46,4.37,2.89,2.51,38.54 15,1,14,2.6,4.35,6.02,5.16,1.46,1.93,3.84,5.57,7.01,5.49,3.85,3.49,50.99 16,2,90,4.26,12.45,7.52,3.75,2.46,3.36,3.66,4.54,8.18,5.84,4.07,3.38,58.09 16,2,14,7.98,11.62,13.01,10.08,11.22,6.87,4.1,7.96,9.21,4.55,5.57,4.43,90.85 17,1,90,0.83,13.55,8.01,7.14,4.58,5.21,7,5.76,3.67,3.83,5.06,3.08,69.27 17,1,14,1.39,2.52,5.04,2.72,1.71,1.26,1.62,1.42,0.98,0.94,1.24,0.75,20.61 18,1,2,0.85,0.45,0.54,0.85,4.05,2,0.84,0.36,0.37,0.31,0.38,0.4,12.22 18,1,90,2.18,9.59,5.99,6.48,4.69,4.93,4.71,3.26,2.6,2.67,3.64,4,53.62 18,1,14,1.58,6.72,3.82,1.82,1.54,1.07,0.92,1.02,0.77,1.32,1.45,1.29,19.14 19,2,2,0.58,0.85,0.66,4.56,2.17,0.99,0.57,0.41,0.35,0.27,0.29,0.32,12.11 19,2,90,1.19,10.47,4.99,2.15,1.52,1.08,3.36,2.49,1.4,0.96,0.84,2.18,29.31 19,2,14,2.48,5.03,7.76,6.76,2.24,2.34,4.38,3.01,2.25,3.5,3.11,1.52,44.36 20,1,90,1.31,1.16,1.79,4.23,7.65,5.22,3.47,2.39,2.7,1.89,1.6,1.66,39.51 21,1,90,0.78,9.27,4.46,2.57,1.72,0.79,0.22,0.32,0.37,1.21,2.55,2.42,19.91

______________________________________________ R-help_at_r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.

Received on Thu 24 Jul 2008 - 15:09:54 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Thu 24 Jul 2008 - 19:32:25 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive