[Rd] Brainstorm: Alpha and Beta testing of R versions

From: Andrew Robinson <A.Robinson_at_ms.unimelb.edu.au>
Date: Sun 06 Nov 2005 - 00:01:30 GMT

Hi Martin,

On Fri, Nov 04, 2005 at 09:58:47AM +0100, Martin Maechler wrote:
> [Mainly for R-foundation members; but kept in public for general
> brainstorming...]

I'll take up the invitation to brainstorm.

As a user of R for a number of years, I'd really like to perform some useful service. I use a relatively obscure platform (FreeBSD) and I can compile code. I'd like to think that I'm in the target market for beta testing :). But, I'm timid. I do not feel, in general, that R core welcomes bug reports.

I think that there are several things that could be tried to encourage more, and more useful, bug reports.

  1. Put the following text on the *front page* of the tracking system, so that it is seen before the reader clicks on "New Bug Report":

"Before submitting a bug report, please read Chapter `R Bugs' of `The R FAQ'. It describes what a bug is and how to report a bug.

If you are not sure whether you have observed a bug or not, it is a good idea to ask on the mailing list R-Help by sending an e-mail to r-help@stat.math.ethz.ch rather than submitting a bug report."

(BTW is this true also for alpha/beta testing?)

2) Try to use the structure of the reporting page to prompt good

   reporting. On the report page, summarize the key points of    identifying and reporting a bug in a checklist format. Maybe even    insist that the boxes be checked before allowing submission.    Include seperate text boxes for description and sample code, to    suggest that sample code is valued.

3) On either or both pages (and in FAQ), explain that thoughtful bug

   reports are valued and appreciated. Further, explain that bug    reports that do not follow the protocol are less valuable, and take    more time.

4) Add checkboxes to the report page for alpha/beta. (I suggest this

   for the purposes of marketing, not organization.)

5) On the report page, include hyperlinks to archived bug reports that

   were good. Do likewise with some artificial bug reports that are    bad.

6) Add an intermediate, draft step for bug submission, to allow

   checking. If possible, include as part of this step an automated    pattern matching call that identifies similarly texted bug reports,    provides links to the reports, and invites a last-minute cross-check.

7) Keep a list of people who report useful bugs in alpha/beta phase on

   the website. Many academics could point to it as evidence of    community service.

> In order to discourage an increased number of non-bug reports we
> may have to also open a "hall of shame" though...

8) I'm sure that you're being ironic! But I will take the point

   seriously, for what it's worth. I think that humiliating    submitters who haven't followed the protocol is deleterious. It    seems like almost every month we see someone get slapped on the    wrist for not doing something the right way. Of course, it's    frustrating that people aren't following the posting guide. But,    why is that? Where is the breakdown? It might be interesting to    try some follow-up (an exit interview!). If someone has failed to    follow the protocol, perhaps we should try to find out why it was    confusing, or if they just ignored it.

   The R-core is surrounded by, and serves, a community that comprises    people who are not sufficiently good at what R-core does to be    invited in to R-core. But, we're clearly interested in what R-core    produces. Please don't assume that bug submissions that do not    follow the R protocol are the consequence of deliberate    malfeasance.

   To paraphrase Ian Fleming: Once is happenstance. Twice is    incompetence. The third time, Mr. Bond, is enemy action. So, ...

9) Publicly thank bug reporters whether their reports are useful or

   not. I just googled 'R-devel thank' and you figure prominently,    Martin :).



Andrew Robinson
Senior Lecturer in Statistics                       Tel: +61-3-8344-9763
Department of Mathematics and Statistics            Fax: +61-3-8344-4599
University of Melbourne, VIC 3010 Australia
Email: a.robinson_at_ms.unimelb.edu.au    Website: http://www.ms.unimelb.edu.au

R-devel@r-project.org mailing list
Received on Sun Nov 06 11:15:41 2005

This archive was generated by hypermail 2.1.8 : Mon 20 Feb 2006 - 03:21:33 GMT