Re: [R] ok to use glht() when interaction is NOT significant?

From: array chip <arrayprofile_at_yahoo.com>
Date: Tue, 08 Mar 2011 16:01:23 -0800

Thank you Andy for sharing your thoughts.

John



From: "Liaw, Andy" <andy_liaw_at_merck.com>

Cc: r-help_at_stat.math.ethz.ch
Sent: Tue, March 8, 2011 7:44:45 AM
Subject: RE: [R] ok to use glht() when interaction is NOT significant?

Just to add my ever depreciating $0.02 USD:

Keep in mind that the significance testing paradigm puts a constraint on false positive rate, and let false negative rate float. What you should consider is whether that makes sense in your situation. All too often this is not carefully considered, and sometimes people will do not-very-kosher things to compensate for the conservativism of the significance testing.

If you want to stay with the formality of the "protected tests", you should first check the overall F-test of the entire model and make sure that's significant before you look at the individual terms in the model.

It's not sufficient for A1 and A2 to be significantly different at B2 and not at B1 to say that there's significant interaction, but that the difference between A1 and A2 at B1 has to be significantly different that that at B2. That's the definition of the interaction in the 2x2 case. If you have a priori interest in the comparison of A1 vs. A2 at B2, then you can test it as a pre-planned contrast and not worry too much about "protection" or multiplicity.

HTH,
Andy

> -----Original Message-----
> From: r-help-bounces_at_r-project.org
> [mailto:r-help-bounces_at_r-project.org] On Behalf Of array chip
> Sent: Tuesday, March 08, 2011 1:31 AM
> To: Bert Gunter
> Cc: r-help_at_stat.math.ethz.ch
> Subject: Re: [R] ok to use glht() when interaction is NOT significant?
>
> Hi Bert, thank you for your thoughtful and humorous comments, :-)
>
> It is scientifically meaningful to do those comparisons, and
> the results of
> these comparisons actually make sense to our hypothesis, i.e.
> one is significant
> at B2 level while the other is not at B1 level. Just
> unfortunately, the overall
> F test for interaction is not significant. I understand
> "formally" one should
> not do these post-hoc comparisons under non-significant
> interaction term. But
> should I really stop comparing under this situation,
> especially when these
> comparisons conform to our hypothesis? I am encouraged to see
> that you said "For
> exploratory purposes, such post hoc comparisons might lead to
> great science".
> However, my concern is these results may not pass reviewers
> when sent out for
> publication.
>
> BTW, I am non-US reader, so I did google "never inhaled". :-)
>
> John
>
>
>
>
> ________________________________
> From: Bert Gunter <gunter.berton_at_gene.com>
>
> Cc: r-help_at_stat.math.ethz.ch
> Sent: Mon, March 7, 2011 9:20:11 PM
> Subject: Re: [R] ok to use glht() when interaction is NOT significant?
>
> Inline below
>
>
> > Hi, let's say I have a simple ANOVA model with 2 factors A
> (level A1 and A2)
> >and
> > B (level B1 and B2) and their interaction:
> >
> > aov(y~A*B, data=dat)
> >
> > It turns out that the interaction term is not significant
> (e.g. P value =
> 0.2),
> > but if I used glht() to compare A1 vs. A2 within each level
> of B, I found that
> > the comparison is not significant when B=B1, but is very
> significant (P<0.01)
> > when B=B2.
> >
> > My question is whether it's legal to do this post-hoc
> comparison when the
> > interaction is NOT significant? Can I still make the claim
> that there is a
> > significant difference between A1 and A2 when B=B2?
>
> (I am serious here). Don't know what "legal" means. Why do you want to
> make the claim? When does it **ever** mean anything scientifically
> meaningful to make it? What is the **scientific** question of
> interest? Are the data unbalanced? Have you plotted the data to tell
> you what's going on?
>
> Warning: I come from the school (maybe I'm the only student...) that
> believes all such formal post hoc comparisons are pointless, silly,
> wastes of effort. Note the word: "formal" -- that is pretending the P
> values mean anything, For exploratory purposes, which can certainly
> include producing P values as well as graphs, such post hoc
> comparisons might lead to great science. It's the "formal" part that I
> reject and that you seem to be hung up on.
>
> Note also: If you're a Bayesian and can put priors on everything, you
> can spit out posteriors and Bayes factors to your heart's content.
> Really! -- no need to sweat multiplicity even. Of course, I speak here
> only as an observer, having never actually inhaled myself.*
>
> Cheers,
> Bert
>
> *Apologies to all non-US and younger readers. This is a smart-aleck
> reference to an infamous dumb remark from a recent famous, smart
> former U.S. president. Google "never inhaled" for details.
>
> >
> > Thanks
> >
> > John
> >
> >
> >
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help_at_r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
>
> --
> Bert Gunter
> Genentech Nonclinical Biostatistics
> 467-7374
> http://devo.gene.com/groups/devo/depts/ncb/home.shtml
>
>
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help_at_r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
Notice: This e-mail message, together with any attachme...{{dropped:16}}



R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Wed 09 Mar 2011 - 00:03:52 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Wed 09 Mar 2011 - 01:00:19 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive