Re: [R] p-values

From: Spencer Graves <spencer.graves_at_pdf.com>
Date: Thu 11 Aug 2005 - 13:01:48 EST

          pperm seems reasonable, though I have not looked at the details.

          We should be careful about terminology, however. So-called "exact p-values" are generally p-values computed assuming a distribution over a finite set of possible outcomes assuming some constraints to make the outcome space finite. For example, Fisher's exact test for a 2x2 table assumes the marginals are fixed.

          I don't remember the details now, but I believe there is literature claiming that this may not be the best thing to do when, for example, when it is reasonable to assume that the number in each cell is Poisson.   In such cases, you may lose statistical power by conditioning on the marginals. I hope someone else will enlighten us both, because I'm not current on the literature in this area.

          The situation with "exact tests" and "exact p-values" is not nearly as bad as with so-called ""exact confidence intervals", which promise to deliver at least the indicated coverage probability. With discrete distributions, it is known that 'Approximate is better than "exact' for interval estimation of binomial proportions', as noted in an article of this title by A. Agresti and B. A. Coull (1998) American Statistician, 52: 119-126. (For more on this particular issue, see Brown, Cai and Dasgupta 2003 "Interval Estimation in Exponential Families", Statistica Sinica 13: 19-49).

          If this does not answer your question adequately, may I suggest you try the posting guide. People report having found answers to difficult questions in the process of preparing a question following that guide, and when they do post a question, they are much more likely to get a useful reply.

          spencer graves

Peter Ho wrote:

> Spencer,
>
>
> Thank you for referring me to your other email on Exact goodness-of-fit
> test. However, I'm not entirely sure if what you mentioned is the same
> for my case. I'm not a statistician and it would help me if you could
> explain what you meant in a little more detail. Perhaps I need to
> explain the problem in more detail.
>
> I am looking for a way to calculate exaxt p-values by Monte Carlo
> Simulation for Durbin's test. Durbin's test statistic is similar to
> Friedman's statistic, but considers the case of Balanced Incomplete
> block designs. I have found a function written by Felipe de Mendiburu
> for calculating Durbin's statistic, which gives the chi-squared p-value.
> I have also been read an article by Torsten Hothorn "On exact rank Tests
> in R" (R News 1(1), 1112.) and he has shown how to calculate Monte
> Carlo p-values using pperm. In the article by Torsten Hothorn he gives:
>
> R> pperm(W, ranks, length(x))
>
> He compares his method to that of StatXact, which is the program Rayner
> and Best suggested using. Is there a way to do this for example for the
> friedman test.
>
> A paper by Joachim Rohmel discusses "The permutation distribution for
> the friendman test" (Computational Statistics & Data Analysis 1997, 26:
> 83-99). This seems to be on the lines of what I need, although I am not
> quite sure. Has anyone tried to recode his APL program for R?
>
> I have tried a number of things, all unsucessful. Searching through
> previous postings have not been very successful either. It seems that
> pperm is the way to go, but I would need help from someone on this.
>
> Any hints on how to continue would be much appreciated.
>
>
> Peter
>
>
> Spencer Graves wrote:
>

>> Hi, Peter:
>>
>>       Please see my reply of a few minutes ago subject:  exact 
>> goodness-of-fit test.  I don't know Rayner and Best, but the same 
>> method, I think, should apply.  spencer graves
>>
>> Peter Ho wrote:
>>
>>  
>>
>>> HI R-users,
>>>
>>> I am trying to repeat an example from Rayner and Best "A contingency 
>>> table approach to nonparametric testing (Chapter 7, Ice cream example).
>>>
>>> In their book they calculate Durbin's statistic, D1, a dispersion 
>>> statistics, D2, and a residual. P-values for each statistic is 
>>> calculated from a chi-square distribution and also Monte Carlo p-values.
>>>
>>> I have found similar p-values based on the chi-square distribution by 
>>> using:
>>>
>>> > pchisq(12, df= 6, lower.tail=F)
>>> [1] 0.0619688
>>> > pchisq(5.1, df= 6, lower.tail=F)
>>> [1] 0.5310529
>>>
>>> Is there a way to calculate the equivalent Monte Carlo p-values?
>>>
>>> The values were 0.02 and 0.138 respectively.
>>>
>>> The use of the approximate chi-square probabilities for Durbin's test 
>>> are considered not good enough according to Van der Laan (The 
>>> American Statistician 1988,42,165-166).
>>>
>>>
>>> Peter
>>> --------------------------------
>>> ESTG-IPVC
>>>
>>> ______________________________________________
>>> R-help@stat.math.ethz.ch mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide! 
>>> http://www.R-project.org/posting-guide.html
>>>   
>>
>>
>>  
>>

>
-- 
Spencer Graves, PhD
Senior Development Engineer
PDF Solutions, Inc.
333 West San Carlos Street Suite 700
San Jose, CA 95110, USA

spencer.graves@pdf.com
www.pdf.com <http://www.pdf.com>
Tel:  408-938-4420
Fax: 408-280-7915

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Received on Thu Aug 11 13:07:29 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:39:45 EST