Re: [R] Lack of independence in anova()

From: Ted Harding <>
Date: Thu 07 Jul 2005 - 04:29:24 EST

On 06-Jul-05 Phillip Good wrote:
> Do you or Lumley have a citation for this conclusion? Most people go
> forward with the ANOV on the basis that the various tests are
> independent.

This hardly needs a citation -- Thomas Lumley's explanation (excerpted below from Doug Bates's mail) is justification enough, and it is succinct and elementary.

However, a place one naturally looks for caveats of this kind is in Kendall & Stuart, and I duly found it (Vol 3, section 35.46 of my 1983 edition). It is essentially exactly the same explanation:

  "However, the tests in the AV tables which we have considered    are never independent tests, for although the various SS in a    table may be independent of each other, all the tests we have    derived use the Residual SS as denominator of the test statistic,    and the various tests must therefore be statistically dependent,    since, e.g., a Residual SS which is (by chance) large will    depress all the values of the test staistics simultaneously."

(And K&S, thorough as they are with citations, do not cite any primary reference for this either!)

However, if the "degrees of freedom" for Residual SS is large, then the amount of random variation in the denominator will be small and it will be effectively constant. Then, of course, with independent numerators, the tests will be effectively independent (and equivalent to chi-squared) and also, therefore, the p-values.

The fact that "most people go forward with the ANOV on the basis that the various tests are independent" possibly reflects the wide-spread act of faith that one has "a large sample", whatever the value of n may really be. One wonders how often people check the p-value for their F on (n1:n2) d.f. against the p-value for (n1:Inf) d.f.? The 5% point decreases quite perceptibly as n2 increases up to about 20, and more slowly thereafter; but still the difference between F(n1:20) and F(n1:Inf) is substantial for any n1 (being about 0.5 for n1 up to about 10, increasing thereafter up to 0.84):

n1<-c(1+2*(0:50),5000);cbind(n1,qf(0.95,n1,20) - qf(0.95,n1,Inf))

F(Inf,Inf) = 1 ; F(20:Inf) = qf(0.95,Inf,20) = 1.841

Conversely (e.g.):

  > 1-pf(qf(0.95,20,20),20,20)
  [1] 0.05
  > 1-pf(qf(0.95,20,20),20,Inf)
  [1] 0.002391189

Such differences are related to the degree of non-independence of several tests on the same data.

> [Douglas Bates]:
> Thomas Lumley came to my rescue with an explanation. There is no
> reason why the results of the F tests should be independent. The
> numerators are independent but the denominator is the same for both
> tests. When, due to random variation, the denominator is small, then
> the p-values for both tests will tend to be small. If, instead of
> F-tests you use chi-square tests then you do see independence.

But surely this amounts to assuming n2 = Inf? If that's an adequate approximation, then fine; but if not (see e.g. above) then not!

Best wishes to all,

E-Mail: (Ted Harding) <> Fax-to-email: +44 (0)870 094 0861
Date: 06-Jul-05                                       Time: 19:29:18
------------------------------ XFMail ------------------------------

______________________________________________ mailing list PLEASE do read the posting guide! Received on Thu Jul 07 05:47:20 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:33:17 EST