Re: [R] logistic discrimination: which chance performance??

From: Bruno L. Giordano <bruno.giordano_at_music.mcgill.ca>
Date: Fri 11 Aug 2006 - 13:07:41 EST

Well,
If posting a possible solution to one's own problem is not part of the netiquette of this list please correct me.

Following Titus et al. (1984) one might use Cohen's kappa to have a chance-corrected measure of agreement between the original and reproduced classification:

Kappa() in library vcd
kappa2() in library irr
ckappa() in library psy
cohen.kappa() in library concord......

    Bruno

Kimberly Titus; James A. Mosher; Byron K. Williams (1984), Chance-corrected Classification for Use in Discriminant Analysis: Ecological Applications, American Midland Naturalist, 111(1),1-7.

> Hello,
> I am using logistic discriminant analysis to check whether a known
> classification Yobs can be predicted by few continuous variables X.
>
> What I do is to predict class probabilities with multinom() in nnet(),
> obtaining a predicted classification Ypred and then compute the percentage
> P(obs) of objects classified the same in Yobs and Ypred.
>
> My problem now is to figure out whether P(obs) is significantly higher
> than
> chance.
>

> I opted for a crude permutation approach: compute P(perm) over 10000
> random
> permutations of Yobs (i.e., refit the multinom() model 10000 times
> randomly
> permuting Yobs) and consider P(obs) as significantly higher than chance if
> higher than the 95th percentile of the P(perm) distribution.
>
> Now, the problem is that the mode of P(perm) is always really close to
> P(obs), e.g., if P(obs)=1 (perfect discrimination) also the most likely
> P(perm) value is 1!!!
>
> I figured out that this is due to the fact that, with my data, randomly
> permuted classifications are highly likely to strongly agree with the
> observed classification Yobs, but, probably since my machine learning
> background is almost 0, I am kind of lost about how to proceed at this
> point.
>
> I would greatly appreciate a comment on this.
>
> Thanks
> Bruno
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Bruno L. Giordano, Ph.D.
> CIRMMT
> Schulich School of Music, McGill University
> 555 Sherbrooke Street West
> Montréal, QC H3A 1E3
> Canada
> http://www.music.mcgill.ca/~bruno/
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Fri Aug 11 13:24:30 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Sat 12 Aug 2006 - 02:20:41 EST.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.