Re: [R] Conservative "ANOVA tables" in lmer

From: Greg Snow <>
Date: Wed 13 Sep 2006 - 16:11:08 GMT


 Douglas Bates wrote:
> Hmm - I'm not sure what confidence interval and what number of levels
> you mean there so I can't comment on that method.
> Suppose we go back to Spencer's example and consider if there is a
> signficant effect for the Nozzle factor. That is equivalent to the
> hypothesis H_0: beta_2 = beta_3 = 0 versus the general alternative. A
> "p-value" could be formulated from an MCMC sample if we assume that
> the marginal distribution of the parameter estimates for beta_2 and
> beta_3 has roughly elliptical contours and you can evaluate that by,
> say, examining a hexbin plot of the values in the MCMC sample. One
> could take the ellipses as defined by the standard errors and
> estimated correlation or, probably better, by the observed standard
> deviations and correlations in the MCMC sample. Then determine the
> proportion of (beta_2, beta_3) pairs in the sample that fall outside
> the ellipse centered at the estimates and with that eccentricity and
> scaling factors that passes through (0,0). That would be an empirical
> p-value for the test.
> I would recommend calculating this for a couple of samples to check on
> the reproducibility.

Here is another thought for an empirical p-value that may be easier to compute and would require fewer assumptions:

Take the proportion of MCMC samples that fall into each quadrant (++, +-, -+, --) and use the smallest of these proportions as the p-value (or the smallest out of a subset of the quadrants for a one-sided style test).

Think of it this way, if the smallest proportion is greater than alpha, then any closed curve (ellipse, polygon, even concave polygons) that includes 1-alpha proportion of the points would need to include points from all 4 quadrants and therefore any convex curve would have to include (0,0) which is consistent with the null hypothesis.

On the other hand if there is a quadrant that contains fewer than alpha percent of the points then there exists at least one confidence region (possibly concave) that contains 1-alpha proportion of the points and excludes (0,0) and that entire quadrant, which is consistent with the alternative that at least one of the betas differs from 0.

A more conservative p-value would be to take the minimum proportion and muliply it by 4 (or 2^p for p simultaneous tests) which is the same idea as multipying by 2 for a 2 sided univariate test and assumes that the confidence regions would exclude similar proportions of points in each direction (central confidence regions rather than minimum length or other confidence regions). This seems to me that it would be over conservative in some cases (since all the proportions must sum to 1, we don't really have 4 degrees of freedom and a smaller adjustment factor may still be correct and less conservative).

Some simulations would be a good idea to see if the plain minimum is to liberal and how conservative the other approach is for common situations.

This is just my first thoughts on the matter, I have not tested anything, so any comments or other discussion of this idea is welcome.

Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
(801) 408-8111

______________________________________________ mailing list
PLEASE do read the posting guide
and provide commented, minimal, self-contained, reproducible code.
Received on Thu Sep 14 02:40:57 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Wed 13 Sep 2006 - 17:30:05 GMT.

Mailing list information is available at Please read the posting guide before posting to the list.