Re: [Rd] R CMD check --force-multiarch does not install all the archs for testing

From: Hervé Pagès <hpages_at_fhcrc.org>
Date: Fri, 15 Jul 2011 16:53:09 -0700

Hi Uwe,

On 11-06-29 10:50 AM, Uwe Ligges wrote:
>
>
> On 28.06.2011 23:11, Hervé Pagès wrote:
>> Simon,
>>
>> On 11-06-28 01:44 PM, Simon Urbanek wrote:
>>>
>>> On Jun 28, 2011, at 3:45 PM, Hervé Pagès wrote:
>>>
>>>> Hi Simon,
>>>>
>>>> On 11-06-28 12:19 PM, Simon Urbanek wrote:
>>>>>
>>>>> On Jun 28, 2011, at 3:01 PM, Hervé Pagès wrote:
>>>>>
>>>>>> Hi Uwe,
>>>>>>
>>>>>> On 11-06-28 01:44 AM, Uwe Ligges wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 28.06.2011 01:31, Hervé Pagès wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Why isn't 'R CMD check --force-multiarch' installing the package
>>>>>>>> for all the architectures that are going to be checked?
>>>>>>>
>>>>>>> Hervé,
>>>>>>>
>>>>>>> no, since it cannot know that that you need
>>>>>>>
>>>>>>> --merge-multiarch
>>>>>>>
>>>>>>> as an additional install flag for this particular package.
>>>>>>
>>>>>> Why not just use this flag anyway? Does it hurt to use it on
>>>>>> packages that don't strictly need it?
>>>>>>
>>>>>
>>>>> It does for two reasons: a) everything is built twice
>>>>
>>>> That's exactly what I want when I do 'R CMD check --force-multiarch'
>>>>
>>>
>>> No, that's not what it does (and I assume you mean --force-biarch). It
>>> builds the package just once, you're simply overriding the default
>>> behavior of checking for configure.win, that's all. The two flags are
>>> orthogonal, --force-biarch makes no sense with --merge-multiarch, they
>>> are for all practical purposes mutually exclusive by definition of
>>> what they do.
>>
>> I really mean --force-multiarch, not --force-biarch. AFAIK 'R CMD check'
>> has no --force-biarch option.
>>
>>>
>>>
>>>>> and b) package authors don't expect the necessity to support
>>>>> --libs-only if the package doesn't require separate build runs.
>>>>
>>>> When specifying --force-multiarch, the user really expects the
>>>> package to be installed for all sub-archs.
>>>>
>>>
>>> ... with the assumption that the package supports it even though it
>>> has a configure script which may to may not work unlike
>>> --merge-multiarch which will always work.
>>
>> Just to clarify:
>>
>> --force-multiarch is an 'R CMD check' option, not an 'R CMD INSTALL'
>> option
>>
>> --merge-multiarch is an 'R CMD INSTALL' option, not an 'R CMD check'
>> option
>>
>> Now you are saying that --merge-multiarch will always work (on Windows
>> of course, all this discussion is about how to achieve multiarch check
>> on Windows). Great, this is what I've been observing too so far!
>
>
> It does not "always work", unfortunately. We need force-biarch on CRAN for:
>
> RGtk2
> rgdal
> rphast
> SQLiteMap
>
>
>
>> On packages with or without native codes, with or without configure.win
>> scripts, etc... It always seems to work. So, again, why isn't
>> 'R CMD check --force-multiarch' installing with --merge-multiarch?
>> Note that I'm not attached to that solution in particular, just trying
>> to suggest an easy fix for 'R CMD check --force-multiarch' (which right
>> now is broken on some packages).
>
> Well, merge-multiarch will result in small overhead. It will try to
> compile the code in ./src for the other architecture and merge that into
> the same package. For packages without any compiled code, it does not
> make sense to ask R to do that since the packages are identical under
> 32-bit and 64-bit, hence you don't want the overhead.

If the package doesn't contain any native code, then trying to compile the native code for the other arch shouldn't result in a big overhead. What's the overhead of an if statement like:

   ## add DLL for x64
   if (file.exists("installation_folder/pkgname/libs") &&

       !file.exists("installation_folder/pkgname/libs/x64")) {
       ...
       ## compile and install DLL for x64
       ...

   }
   ## test if installed package can be loaded for x64    ...

Note that the "add DLL for x64" step is currently doing something *very* useful, which is to make sure that the package can be loaded for this arch:

D:\biocbld\bbs-2.9-bioc\meat>..\R\bin\R CMD INSTALL --merge-multiarch BSgenome_1.21.3.tar.gz

install for i386

Attaching package: 'IRanges'

The following object(s) are masked from 'package:base':

     Map, cbind, eval, intersect, mapply, order, paste, pmax, pmax.int,
     pmin, pmin.int, rbind, rep.int, setdiff, table, union

** help
*** installing help indices

** building package indices ...
*** tangling vignette sources ...

    'BSgenomeForge.Rnw'
    'GenomeSearching.Rnw'
** testing if installed package can be loaded

add DLL for x64

It's useful even for packages without compiled code since they can depend on packages that do have compiled code but have been installed only for 1 arch.

Unfortunately this is something that --force-biarch doesn't do (it returns success even if compilation for x64 failed).

> For packages that
> include compiled code, you still do not want it, since R does it
> automatically - unless there is a configure.win file.

I want to be able to use the same command for any package. 'R CMD check' looses *a lot* of value if people need to know what's in a package in order to choose the right command for doing a multiarch check. I'm not talking about special configuration options that some packages with compiled code need. I'm talking about the situation where everything is setup properly so the 2 separate checks work:

   R --arch i386 CMD check pkg_x.y.z.tar.gz    R --arch x64 CMD check pkg_x.y.z.tar.gz

but 'R CMD check --force-multi-arch pkg_x.y.z.tar.gz' doesn't because it fails to install for x64.

So if installing with --merge-multiarch is "almost" the right thing to do (even though it works for only 100% of the BioC packages and 99.8% of the CRAN packages -- note that this percentage would drop without using --merge-multiarch), then I do want to use it because this is the closest I can get to *the* command that works for any package.

Another reason I want to use it for any package that includes compiled code, even for packages with a configure.win file, is because "R automatic bi-arch installation" feature will report success even if compilation for x64 failed.

>
> Therefore, the option is just an add on that is necessary for only 18
> out of roughly 3000 CRAN packages.
>
> So we have 4 packages that require --force-biarch and 18 requiring
> --merge-multiarch on CRAN. All the others build automatically for both
> architectures if they need to. So no reason to do that all the time.

For our BioC builds, we are lucky that --merge-multiarch works for all our software packages. Unfortunately we cannot just use

   R CMD check --install-args="--merge-multiarch"

because 'R CMD check' calls 'R CMD INSTALL --merge-multiarch' on the *extracted* tarball but --merge-multiarch only works on a *source* tarball. So we use this long and ugly command (that is basically the 2-step command that you provided earlier in this thread, thanks!):  

http://bioconductor.org/checkResults/2.9/bioc-LATEST/Biobase/moscato1-checksrc.html

Anyway I still think anybody should be able to run a multiarch check. Not only the CRAN, or R-forge, or BioC build systems. It should be easy (1 step command). It should not need any "add-on".

Thanks,
H.

>
> Best wishes,
> Uwe
>
>
>
>>> --force-biarch is just a way to flag packages that have configure.win
>>> that has no effect on the binary settings (flags etc.). It forces R to
>>> try multi-arch build in one flight, but it may or may not work
>>> depending on the package. It is a way to save time by not running
>>> --merge-multiarch (and thus building the package twice).
>>
>> --force-biarch is an 'R CMD INSTALL' option that I don't use. Why would
>> I use something that might fail when I can use --merge-multiarch which
>> always works.
> >
>> Thanks,
>> H.
>>
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>>>>
>>>>> The cross-platform way is to not use --merge-multiarch but use
>>>>> --libs-only instead as needed (easy to check after the first arch
>>>>> run which will tell you whether it's needed or not). I suspect that
>>>>> --merge-multiarch is just a convenience shortcut (and it's unclear
>>>>> to me why it's Windows-only...).
>>>>
>>>> A great convenience indeed as it allows to do the multiarch install in
>>>> a single step. And it's unclear to me too why it's Windows-only but I
>>>> would have hoped you would know...
>>>>
>>>> Thanks,
>>>> H.
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>> Simon
>>>>>
>>>>>
>>>>>>> You will have to check it in repository maintainer's mode (as the
>>>>>>> CRAN
>>>>>>> maintainers do everywhere). Essentially this is for me (when also
>>>>>>> producing WIndows binaries):
>>>>>>>
>>>>>>>
>>>>>>> Step 1: Installation
>>>>>>>
>>>>>>> R CMD INSTALL --pkglock --compact-docs --build --merge-multiarch
>>>>>>> --library="D:/path/to/library" fabia_1.5.0.tar.gz>
>>>>>>> fabia-install.out 2>&1
>>>>>>>
>>>>>>> (where the merge-multiarch part applies only to this package, of
>>>>>>> course)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Step 2: Check (without installation, since that happened before
>>>>>>> already,
>>>>>>> using the install log from step 1)
>>>>>>>
>>>>>>> R CMD check --library="D:/path/to/library" --force-multiarch
>>>>>>> --install="check:fabia-install.out" fabia
>>>>>>
>>>>>> Whaoooo! Would be nice if there was a plan to make 'R CMD check' also
>>>>>> usable by normal people (including the package developer), not just
>>>>>> by a few privileged people that know about those undocumented tricks.
>>>>>>
>>>>>> Thanks,
>>>>>> H.
>>>>>>
>>>>>>>
>>>>>>> Best wishes,
>>>>>>> Uwe
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> For some packages, it only installs for the default arch ('i386').
>>>>>>>> Then testing the package for 'x64' fails.
>>>>>>>>
>>>>>>>> For example,
>>>>>>>>
>>>>>>>> Output of R CMD check --force-multiarch fabia_1.5.0.tar.gz:
>>>>>>>> -----------------------------------------------------------
>>>>>>>> * using log directory 'D:/biocbld/bbs-2.9-bioc/meat/fabia.Rcheck'
>>>>>>>> * using R version 2.14.0 Under development (unstable) (2011-05-30
>>>>>>>> r56020)
>>>>>>>> * using platform: i386-pc-mingw32 (32-bit)
>>>>>>>> * using session charset: ISO8859-1
>>>>>>>> * using option '--no-vignettes'
>>>>>>>> * checking for file 'fabia/DESCRIPTION' ... OK
>>>>>>>> * this is package 'fabia' version '1.5.0'
>>>>>>>> * checking package name space information ... OK
>>>>>>>> * checking package dependencies ... OK
>>>>>>>> * checking if this is a source package ... OK
>>>>>>>> * checking whether package 'fabia' can be installed ... OK
>>>>>>>> * checking installed package size ... OK
>>>>>>>> * checking package directory ... OK
>>>>>>>> * checking for portable file names ... OK
>>>>>>>> * checking DESCRIPTION meta-information ... OK
>>>>>>>> * checking top-level files ... OK
>>>>>>>> * checking index information ... OK
>>>>>>>> * checking package subdirectories ... OK
>>>>>>>> * checking R files for non-ASCII characters ... OK
>>>>>>>> * checking R files for syntax errors ... OK
>>>>>>>> * loading checks for arch 'i386'
>>>>>>>> ** checking whether the package can be loaded ... OK
>>>>>>>> ** checking whether the package can be loaded with stated
>>>>>>>> dependencies
>>>>>>>> ... OK
>>>>>>>> ** checking whether the package can be unloaded cleanly ... OK
>>>>>>>> ** checking whether the name space can be loaded with stated
>>>>>>>> dependencies ... OK
>>>>>>>> ** checking whether the name space can be unloaded cleanly ... OK
>>>>>>>> * loading checks for arch 'x64'
>>>>>>>> ** checking whether the package can be loaded ...Warning: running
>>>>>>>> command '"D:/biocbld/bbs-2.9-bioc/R/bin/x64/Rterm.exe"
>>>>>>>> R_ENVIRON_USER='no_such_file' --no-site-file --no-init-file
>>>>>>>> --no-save
>>>>>>>> --no-restore --slave -f
>>>>>>>> D:\biocbld\bbs-2.9-bioc\tmpdir\RtmpO65p5H\Rin57456988' had status 1
>>>>>>>> ERROR
>>>>>>>> Error: package 'fabia' is not installed for 'arch=x64'
>>>>>>>> Execution halted
>>>>>>>>
>>>>>>>> It looks like this package has a loading problem: see the
>>>>>>>> messages for
>>>>>>>> details.
>>>>>>>>
>>>>>>>> Content of fabia.Rcheck\00install.out:
>>>>>>>> --------------------------------------
>>>>>>>>
>>>>>>>> * installing *source* package 'fabia' ...
>>>>>>>> Building libRcpp.a in RcppSrc...
>>>>>>>> rm -f Rcpp.o libRcpp.a
>>>>>>>> g++ -c Rcpp.cpp -o Rcpp.o -I"D:/biocbld/BBS-2˜1.9-B/R/include"
>>>>>>>> -I"D:/biocbld/BBS-2˜1.9-B/R/src/include" -Wall -O2
>>>>>>>> ar r libRcpp.a Rcpp.o
>>>>>>>> C:\Rtools213\MinGW\bin\ar.exe: creating libRcpp.a
>>>>>>>> ranlib libRcpp.a
>>>>>>>> rm -f Rcpp.o
>>>>>>>> rm -f Rcpp.o
>>>>>>>> ** libs
>>>>>>>> running src/Makefile.win ...
>>>>>>>> rm -f fabia.o fabia.dll *.a *.o *.so *.dll
>>>>>>>> g++ -c fabiac.cpp -o fabia.o -I../RcppSrc
>>>>>>>> -I"D:/biocbld/BBS-2˜1.9-B/R/include" -Wall -O2
>>>>>>>> g++ -shared -s -static-libgcc fabia.o -L../RcppSrc -lRcpp
>>>>>>>> -L"D:/biocbld/BBS-2˜1.9-B/R/bin/i386" -lR -o fabia.dll
>>>>>>>> rm -f fabia.o *.a *.o *.so
>>>>>>>> installing to
>>>>>>>> D:/biocbld/bbs-2.9-bioc/meat/fabia.Rcheck/fabia/libs/i386
>>>>>>>> ** R
>>>>>>>> ** demo
>>>>>>>> ** inst
>>>>>>>> ** preparing package for lazy loading
>>>>>>>> Creating a generic function for "plot" from package "graphics" in
>>>>>>>> package "fabia"
>>>>>>>> ** help
>>>>>>>> *** installing help indices
>>>>>>>> ** building package indices ...
>>>>>>>> *** tangling vignette sources ...
>>>>>>>> 'fabia.Rnw'
>>>>>>>> ** testing if installed package can be loaded
>>>>>>>>
>>>>>>>> * DONE (fabia)
>>>>>>>>
>>>>>>>> The source tarball for this package is available here:
>>>>>>>> http://bioconductor.org/packages/2.9/bioc/html/fabia.html
>>>>>>>>
>>>>>>>> What command should be used to perform a multiarch check of this
>>>>>>>> package?
>>>>>>>>
>>>>>>>> This is on a 64-bit Windows Server 2008 R2 Enterprise machine
>>>>>>>> using a
>>>>>>>> recent combined Windows 32/64 bit binary of R-devel from CRAN.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> H.
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Hervé Pagès
>>>>>>
>>>>>> Program in Computational Biology
>>>>>> Division of Public Health Sciences
>>>>>> Fred Hutchinson Cancer Research Center
>>>>>> 1100 Fairview Ave. N, M1-B514
>>>>>> P.O. Box 19024
>>>>>> Seattle, WA 98109-1024
>>>>>>
>>>>>> E-mail: hpages_at_fhcrc.org
>>>>>> Phone: (206) 667-5791
>>>>>> Fax: (206) 667-1319
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel_at_r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Hervé Pagès
>>>>
>>>> Program in Computational Biology
>>>> Division of Public Health Sciences
>>>> Fred Hutchinson Cancer Research Center
>>>> 1100 Fairview Ave. N, M1-B514
>>>> P.O. Box 19024
>>>> Seattle, WA 98109-1024
>>>>
>>>> E-mail: hpages_at_fhcrc.org
>>>> Phone: (206) 667-5791
>>>> Fax: (206) 667-1319
>>>>
>>>>
>>>
>>
>>

-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages_at_fhcrc.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

______________________________________________
R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Fri 15 Jul 2011 - 23:55:53 GMT

This quarter's messages: by month, or sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

All messages

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Sun 17 Jul 2011 - 02:20:10 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.

list of date sections of archive