[Rd] Re: package checks and tests; was: Re: A "rude" question

From: Paul Gilbert <pgilbert_at_bank-banque-canada.ca>
Date: Thu 03 Feb 2005 - 01:58:01 EST

[moved to R-devel]

I am a bit concerned by the fact that package developers would be removing all tests. The tests seem like one of the main mechanisms for ensuring that CRAN does not descend to the almost useless state of statlib: there is an automatic mechanism for removing packages that no longer work.

I have several tests that I don't put in packages, because they are very long or they don't work without locally available databases, but I think it is important that there should be tests that check the main functionality of packages. For very long tests like monte carlo simulations it is usually fairly easy to devise some short version (a few iterations) which checks that the software actually works and gives results that are the same as they always have been, even though it does not check the more serious statistical problems one would actually want to do.

Perhaps there should be another directory, called something like longTests, that is not run daily but is run occasionally.

On r-help I suggested that users might contribute tests to package developers, but another possibility is that there is some mechanism for them to contribute suites of package tests directly to CRAN. This would provide a mechanism for more complete checking of packages, and perhaps a just system for removing packages when the developer is no longer active and the package is broken.

Paul Gilbert

Uwe Ligges wrote:
> Paulo Justiniano Ribeiro Jr wrote:
>

>> One problem with distributing packages with the test sub-directory
>> is that this can overload the daily tests in the CRAN machines
>>
>> My workaround for that  is:
>> For the geoR package I run the tests in my machine but remove the tests
>> directory when submitting to CRAN.
>> Also I mantain a package web-page where I make the version
>> with tests available for downloading

>
>
> [moved to R-devel, chosen a more sensible subject]
>
> If tests are running in a reasonable amount of time, it does not make
> sense to remove them. People want to install the package on a huge
> number of platforms you probably have not tested the package on
> yourself. The only solution is to distribute the tests as well.
>
> Overloading the daily tests on CRAN is another issue (strong related,
> though). Installing and checking (in maintainer's mode, i.e. without
> double-installing) all CRAN packages under R-release for Windows takes 8
> hours on a (dual, but using only one) Xeon 3.06GHz machine these days.
> The checks (without isntallation) of the 15 packages with most intensive
> tests take ~1 hour.
>
> Uwe Ligges
>
>
>
>
>
>> P.J.
>>
>> On Tue, 1 Feb 2005, Spencer Graves wrote:
>>
>>
>>> Hi, Paul:
>>>
>>>      How can I access "the package tests directory" you mentioned?
>>> Only one of the 52 subdirectories of "library" in my current
>>> installation of R 2.0.1 has a "test" folder.
>>>
>>>      Thanks,
>>>      Spencer Graves
>>>
>>> Paul Gilbert wrote:
>>>
>>>
>>>> One point that did not get mentioned in this discussion, and I believe
>>>> deserves
>>>> much more publicity, is the impact of packages tests. The design of
>>>> the package
>>>> system allows package developers to put tests in packages, and these
>>>> are checked
>>>> regularly (see 
>>>> <http://cran.at.r-project.org/contrib/checkSummary.html>).
>>>>
>>>> These are intended to test the package functionality, but also give R
>>>> what is
>>>> perhaps the largest test suite of any statistical software (certainly
>>>> the most
>>>> quickly growing). While any single package's test will never guarantee
>>>> that
>>>> the package works perfectly, the ensemble goes a long way toward 
>>>> ensuring
>>>> that core R functionality behaves as intended. It seems unlikely to me
>>>> that any
>>>> commercial effort will ever be able to catch up.
>>>>
>>>> There are several ways that tests can add to our confidence that
>>>> calculations can be trusted. They can
>>>>  - check against theoretical results
>>>>  - check against published results
>>>>  - check against results from other software
>>>>  - check that calculations done in different ways give the same result
>>>>  - check that monte carlo experiments give distributions that are
>>>> consistent
>>>>     with expected results
>>>>
>>>> Some of these are relatively time consuming to set up and check the
>>>> first time,
>>>> but after that they can be automatic.
>>>>
>>>> If you have particular calculations with specific packages that you
>>>> are especially
>>>> concerned about, I encourage you to participate by devising good tests
>>>> and sending
>>>> them to the package developers. (But first check the tests they are
>>>> already doing
>>>> in the package tests directory.)
>>>>
>>>> Paul Gilbert
>>>>
>>>> msck9@mizzou.edu wrote:
>>>>
>>>>
>>>>> Dear all, I am beginner using R. I have a question about it. When you
>>>>> use it,
>>>>> since it is written by so many authors, how do you know that the
>>>>> results are trustable?(I don't want to affend anyone, also I trust
>>>>> people). But I think this should be a question.
>>>>>
>>>>> Thanks,
>>>>> Ming
>>>>>
>>>>> ______________________________________________
>>>>> R-help@stat.math.ethz.ch mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>> PLEASE do read the posting guide!
>>>>> http://www.R-project.org/posting-guide.html
>>>>>
>>>>>
>>>>>
>>>>
>>>> ______________________________________________
>>>> R-help@stat.math.ethz.ch mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide!
>>>> http://www.R-project.org/posting-guide.html
>>>
>>>
>>> ______________________________________________
>>> R-help@stat.math.ethz.ch mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide! 
>>> http://www.R-project.org/posting-guide.html
>>>
>>>
>>
>>
>> Paulo Justiniano Ribeiro Jr
>> LEG (Laboratório de Estatística e Geoinformação)
>> Departamento de Estatística
>> Universidade Federal do Paraná
>> Caixa Postal 19.081
>> CEP 81.531-990
>> Curitiba, PR  -  Brasil
>> Tel: (+55) 41 361 3573
>> Fax: (+55) 41 361 3141
>> e-mail: paulojus@est.ufpr.br
>> http://www.est.ufpr.br/~paulojus
>>
>> ______________________________________________
>> R-help@stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide! 
>> http://www.R-project.org/posting-guide.html

>
>

>

R-devel@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Thu Feb 03 01:05:59 2005

This archive was generated by hypermail 2.1.8 : Thu 03 Feb 2005 - 01:25:00 EST