Re: [R] Partek has Dunn-Sidak Multiple Test Correction. Is this the same/similar to any of R's p.adjust.methods?

From: Peter Dalgaard <p.dalgaard_at_biostat.ku.dk>
Date: Fri 15 Jul 2005 - 04:46:07 EST

"Earl F. Glynn" <efg@stowers-institute.org> writes:

> The Partek package (www.partek.com) allows only two selections for Multiple
> Test Correction: Bonferroni and Dunn-Sidak. Can anyone suggest why Partek
> implemented Dunn-Sidak and not the other methods that R has? Is there any
> particular advantage to the Dunn-Sidak method?
> R knows about these methods (in R 2.1.1):
>
> > p.adjust.methods
> [1] "holm" "hochberg" "hommel" "bonferroni" "BH" "BY" "fdr"
> [8] "none"
>
> BH is Benjamini & Hochberg (1995) and is also called "fdr" (I wish R's
> documentation said this clearly). BY is Benjamini & Yekutieli (2001).
>
> I found a few hits from Google on Dunn-Sidak, but I'm curious if anyone can
> tell me on a "conservative-liberal" scale, where the Dunn-Sidak method
> falls? My guess is it's less conservative than Bonferroni (but aren't all
> the other methods?), but how does it compare to the other methods?

As far as I gather, D-S is exact for independent tests, conservative for comparisons of group means, and liberal for mutually exclusive tests (in which case Bonferroni is exact). It is always less conservative than Bonferroni, but the difference is small for typical significance levels: when the Bonferroni level is p, the D-S level is

   1 - (1-p/N)^N

and if you put p=0.05 and vary N you'll find that it varies from 0.05 at N=1 down to 0.04877 at N=100000. (Exercise for the students: what is the limit as N goes to infinity?)

The three H-methods play a somewhat different game, basically by only requiring multiple-testing adjustment for non-significant tests. The FDR methods play yet differently by allowing the per test level to increase with the number of significant tests.  

> A limited numerical experiment suggested this order to me: bonferroni (most
> conservative), hochberg and holm about the same, BY, BH (also called fdr),
> and then none.
>
> Thanks for any of thoughts on this.

I'd expect the differences to be fairly small in scenarios where the global null hypothesis is true (excluding "none"). The main difference comes in when some of the nulls are actually false. Also, it depends on your definitions: With the exception of "BY" and "none" the p.adjust methods agree on the smallest adjusted p value, so have the same familywise error rate under the global null. If you count the total number of rejected tests, then you get a difference due to "cascading" in the non-bonferroni cases.

-- 
   O__  ---- Peter Dalgaard             ุster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard@biostat.ku.dk)                  FAX: (+45) 35327907

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Received on Fri Jul 15 04:52:08 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:33:42 EST