Re: [R] About Mcneil Hanley test for a portion of AUC!

From: Frank E Harrell Jr <f.harrell_at_vanderbilt.edu>
Date: Thu, 12 Jun 2008 11:27:02 -0500

Dukka k.c. wrote:
> Dear all
>
> I am trying to compare the performances of several methods using the AUC0.1
> and
> not the whole AUC. (meaning I wanted to compare to AUC's whose x axis only
> goes to
> 0.1 not 1)
>
> I came to know about the Mcneil Hanley test from Bernardo Rangel Tura
> and I referred to the original paper for the calculation of "r" which is an
> argument of the function
> cROC. I can only find the value of "r" for the whole AUC's .
>
>
>
>

>> seROC<-function(AUC,na,nn){
>> a<-AUC
>> q1<-a/(2-a)
>> q2<-(2*a^2)/(1+a)
>> se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
>> se
>> }
>>
>> cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
>> se1<-seROC(AUC1,na1,nn1)
>> se2<-seROC(AUC2,na2,nn2)
>>
>> sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
>> zad<-(AUC1-AUC2)/sed
>> p<-dnorm(zad)
>> a<-list(zad,p)
>> a

>
> Could somebody kindly suggest me how to calculate the value of "r" or
> some ways to calculate the statistical significance measure for the
> differences of auc for a part of
> the curve like AUC0.1.
>
> Thank You

The ROC area is not a sensitive enough measure for comparing two competing predictors. Its power is too low. See for example the following papers. Note that Pencina et al's approach is now in the Hmisc package (function improveProb; documentation to be coming soon). Likelihood ratio tests are even more powerful.

@Article{pen08eva,

   author = {Pencina, Michael J. and {D'Agostino Sr}, Ralph B. and {D'Agostino Jr}, Ralph B. and Vasan, Ramachandran S.},

   title = {Evaluating the added predictive ability of a new marker: {From} area under the {ROC} curve to reclassification and beyond},

   journal = 	 Stat in Med,
   year = 		 2008,
   volume =		 27,
   pages =        {157-172},
   annote =		 {discrimination;model performance;AUC;C-index;risk 
prediction;biomarker;small differences in ROC area can still be very meaningful;example of insignificant test for difference in ROC areas with very significant results from new method;Yates' discrimination slope;reclassification table;limiting version of this based on whether and amount by which probabilities rise for events and lower for non-events when compare new model to old;comparing two models} }

@Article{coo07use,

   author = 		 {Cook, Nancy R.},
   title = 		 {Use and misues of the receiver operating characteristic 
curve in risk prediction},
   journal = 	 {Circulation},
   year = 		 2007,
   volume =		 115,
   pages =		 {928-935},
   annote =		 {reclassification table;problems with c index;problems 
with ROC area;example of large change in predicted risk in cardiovascular disease with tiny change in ROC area;possible limits to c index when calibration is perfect;importance of calibration accuracy and changes in predicted risk when new variables are added} }
-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University

______________________________________________
R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Received on Thu 12 Jun 2008 - 16:40:11 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Thu 12 Jun 2008 - 17:30:42 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive