Re: [Rd] cleanup and Makevars

From: Kasper Daniel Hansen <khansen_at_stat.berkeley.edu>
Date: Wed, 11 Jul 2007 11:32:48 -0700

On Jul 11, 2007, at 9:30 AM, Prof Brian Ripley wrote:

> On Mon, 9 Jul 2007, Kasper Daniel Hansen wrote:
>
>> Hi
>>
>> This is a question prompted by the mac version of R, but as I see it,
>> it should have broader interest.
>>
>> These days the CRAN Mac binary per default compiles every package for
>> two architectures. First i386 and then ppc. In between the two
>> compilation runs, any object files in pkgname/src is removed. This
>> cleanup is necessary since otherwise Make would not recompile the
>> object files under the next architecture.
>>
>> I have a package which includes a large SDK which I have put in
>> src/fusion_sdk
>> (and subdirectories). I take care of that by using a Makevars file
>> with essentially (the whole Makevars file is reproduced below)
>>
>> PKG_SOURCES = \
>> fusion_sdk/calvin_files/data/src/CDFData.cpp
>>
>> OBJS=$(PKG_SOURCES:.cpp=.o)
>>
>> all: $(SHLIB)
>>
>> However, this setup does not remove the object files in the
>> fusion_sdk subdirectory. From a posting on R-sig-mac by Simon Urbanek
>> I learned that I need to clean up these directories myself.
>>
>> I have - per R extensions - attempted to do so using a pkgname/
>> cleanup script. While this works for cleaning up the sub directories
>> when I do R CMD build, it seems as if this script is not being run
>> between compilation runs (is this intentional btw.?).
>
> What do you mean by 'compilation runs'? The 'cleanup' script will
> be run by R CMD INSTALL --cleanup (and without --cleanup it is
> intentionally not run).
>
>> So my question is now: how do I in a simple way clean up my
>> subdirectories? I would prefer it to be as simple as possible because
>> so far I have not really needed anything besides a Makevars file. In
>> fact the impression I have right now from looking at the SHLIB and
>> INSTALL scripts is that I might need a Makefile that removes the
>> object files _before_ starting the compilation, as I don't really see
>> any cleanup process (eg. making a target clean) - but this may very
>> well be due to my limited understanding of these scripts.
>
> I see R CMD INSTALL --clean that runs a clean up afterwards, and
> 2.6.0 will also have R CMD INSTALL --preclean in case you forgot --
> clean on the last run.
>
>> Any help/hints on how to proceed? Especially if I want it to be
>> portable?
>
> What does R CMD INSTALL --clean not do that you want?
>
> If you are talking about R CMD SHLIB, that currently does not clean
> up but this has been raised and it will most likely have a --clean
> option in 2.6.0 that removes $(OBJECTS).

Perhaps I was not clear enough: The CRAN binary of R for Mac has two architectures inside $R_HOME/bin/exec, namely "ppc" and "i386". This means that whenever anyone installs a source package I see two compilation runs - the following is a shortened output:

(Now a lot of other files gets compiled)

g++-4.0 -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - dynamiclib -Wl,-macosx_version_min -Wl,10.3 -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/lib -o affxparser.so fusion_sdk/calvin_files/data/src/CDFData.o fusion_sdk/ calvin_files/data/src/CDFProbeGroupInformation.o fusion_sdk/ calvin_files/data/src/CDFProbeInformation.o fusion_sdk/calvin_files/ data/src/CDFProbeSetInformation.o fusion_sdk/calvin_files/data/src/ CDFQCProbeInformation.o fusion_sdk/calvin_files/data/src/

CDFQCProbeSetInformation.o fusion_sdk/calvin_files/data/src/CELData.o  
fusion_sdk/calvin_files/data/src/DataGroup.o fusion_sdk/calvin_files/ 
data/src/DataGroupHeader.o fusion_sdk/calvin_files/data/src/DataSet.o  
fusion_sdk/calvin_files/data/src/DataSetHeader.o fusion_sdk/ calvin_files/data/src/FileHeader.o fusion_sdk/calvin_files/data/src/
GenericData.o fusion_sdk/calvin_files/data/src/GenericDataHeader.o  
fusion_sdk/calvin_files/exception/src/ExceptionBase.o fusion_sdk/ 
calvin_files/fusion/src/CalvinAdapter/CalvinCELDataAdapter.o  
fusion_sdk/calvin_files/fusion/src/FusionBPMAPData.o fusion_sdk/ calvin_files/fusion/src/FusionCDFData.o fusion_sdk/calvin_files/ fusion/src/FusionCDFQCProbeSetNames.o fusion_sdk/calvin_files/fusion/ src/FusionCELData.o fusion_sdk/calvin_files/fusion/src/GCOSAdapter/ GCOSCELDataAdapter.o fusion_sdk/calvin_files/parameter/src/ ParameterNameValueType.o fusion_sdk/calvin_files/parsers/src/ CDFFileReader.o fusion_sdk/calvin_files/parsers/src/CelFileReader.o
fusion_sdk/calvin_files/parsers/src/DataGroupHeaderReader.o  
fusion_sdk/calvin_files/parsers/src/DataGroupReader.o fusion_sdk/ 
calvin_files/parsers/src/DataSetHeaderReader.o fusion_sdk/ 
calvin_files/parsers/src/DataSetReader.o fusion_sdk/calvin_files/ 
parsers/src/FileHeaderReader.o fusion_sdk/calvin_files/parsers/src/ FileInput.o fusion_sdk/calvin_files/parsers/src/ GenericDataHeaderReader.o fusion_sdk/calvin_files/parsers/src/ GenericFileReader.o fusion_sdk/calvin_files/utils/src/ AffymetrixGuid.o fusion_sdk/calvin_files/utils/src/DateTime.o fusion_sdk/calvin_files/utils/src/FileUtils.o fusion_sdk/calvin_files/ utils/src/StringUtils.o fusion_sdk/calvin_files/utils/src/checksum.o
fusion_sdk/file/BPMAPFileData.o fusion_sdk/file/BPMAPFileWriter.o  
fusion_sdk/file/CDFFileData.o fusion_sdk/file/CELFileData.o  
fusion_sdk/file/FileIO.o fusion_sdk/file/FileWriter.o  
R_affx_cel_parser.o R_affx_cdf_parser.o R_affx_cdf_extras.o  
R_affx_bpmap_parser.o   -F/Library/Frameworks/R.framework/.. - 
framework R
** arch - ppc
g++-4.0 -arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -no-cpp- 
precomp -I/Library/Frameworks/R.framework/Resources/include -I/ 
Library/Frameworks/R.framework/Resources/include/ppc -imacros R_affx_constants.h -Ifusion_sdk/calvin_files/array/src -Ifusion_sdk/ calvin_files/data/src -Ifusion_sdk/calvin_files/exception/src - Ifusion_sdk/calvin_files/fusion/src -Ifusion_sdk/calvin_files/fusion/ src/GCOSAdapter -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter - Ifusion_sdk/calvin_files/parameter/src -Ifusion_sdk/calvin_files/ parsers/src -Ifusion_sdk/calvin_files/portability/src -Ifusion_sdk/ calvin_files/template/src -Ifusion_sdk/calvin_files/utils/src - Ifusion_sdk/calvin_files/writers/src -Ifusion_sdk/file -Ifusion_sdk/ portability -D_USE_MEM_MAPPING_ -I/usr/local/include -fPIC -g -O2 -Wall -O0 -c R_affx_cel_parser.cpp -o R_affx_cel_parser.o

As you see from this (and sorry for all the include statements), when R starts compiling for arch = "ppc", it starts with a with in /src whereas the first compilation run (or more precise the first architecture) starts from /src/fusion_sdk/calvin_files/data/src. This is of course because the object files in the subdirectories do not get removed between the two architectures.

This has - probably - something to do with R CMD SHLIB not cleaning up after itself. A method of removing $(OBJECTS) would be exactly what I need, and may I suggest that this is done per default instead of being set by an option. Anyone using the CRAN binary for the Mac will have the same problem as I.

I am happy to see that this may be fixed for R-2.6.0. Is there anything I can do in the meantime for getting my package to work under R-2.5.1? (I can always move my source files from the /src/ fusion_sdk subdirectories directly into /src, but I would prefer to avoid that).

>>
>> Kasper
>>
>> The full Makevars file:
>>
>> MYCXXFLAGS=-O0
>
> You do realize that is highly non-portable?

Yes, but I do not know of any other way to downgrade the optimization level. The SDK I am using breaks (due to memory alignment issues if a specific optimization flag for GCC is include - the flag is certainly implied by -O2 and as far as I recall also -O1). I guess the solution is to switch to an autoconfigure script. The main reason for not doing this is 1) time and actually more 2) the fact that the SDK does not come with an ac script.

Thanks for the feedback, Kasper

>> %.o: %.cpp
>> $(CXX) $(ALL_CPPFLAGS) $(ALL_CXXFLAGS) $(MYCXXFLAGS) -c $< -o $@
>>
>> PKG_CPPFLAGS=\
>> -imacros R_affx_constants.h\
>> -Ifusion_sdk/calvin_files/array/src\
>> -Ifusion_sdk/calvin_files/data/src\
>> -Ifusion_sdk/calvin_files/exception/src\
>> -Ifusion_sdk/calvin_files/fusion/src\
>> -Ifusion_sdk/calvin_files/fusion/src/GCOSAdapter\
>> -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter\
>> -Ifusion_sdk/calvin_files/parameter/src\
>> -Ifusion_sdk/calvin_files/parsers/src\
>> -Ifusion_sdk/calvin_files/portability/src\
>> -Ifusion_sdk/calvin_files/template/src\
>> -Ifusion_sdk/calvin_files/utils/src\
>> -Ifusion_sdk/calvin_files/writers/src\
>> -Ifusion_sdk/file\
>> -Ifusion_sdk/portability\
>> -D_USE_MEM_MAPPING_
>>
>> PKG_SOURCES = \
>> fusion_sdk/calvin_files/data/src/CDFData.cpp\
>> fusion_sdk/calvin_files/data/src/CDFProbeGroupInformation.cpp\
>> fusion_sdk/calvin_files/data/src/CDFProbeInformation.cpp\
>> fusion_sdk/calvin_files/data/src/CDFProbeSetInformation.cpp\
>> fusion_sdk/calvin_files/data/src/CDFQCProbeInformation.cpp\
>> fusion_sdk/calvin_files/data/src/CDFQCProbeSetInformation.cpp\
>> fusion_sdk/calvin_files/data/src/CELData.cpp\
>> fusion_sdk/calvin_files/data/src/DataGroup.cpp\
>> fusion_sdk/calvin_files/data/src/DataGroupHeader.cpp\
>> fusion_sdk/calvin_files/data/src/DataSet.cpp\
>> fusion_sdk/calvin_files/data/src/DataSetHeader.cpp\
>> fusion_sdk/calvin_files/data/src/FileHeader.cpp\
>> fusion_sdk/calvin_files/data/src/GenericData.cpp\
>> fusion_sdk/calvin_files/data/src/GenericDataHeader.cpp\
>> fusion_sdk/calvin_files/exception/src/ExceptionBase.cpp\
>> fusion_sdk/calvin_files/fusion/src/CalvinAdapter/
>> CalvinCELDataAdapter.cpp\
>> fusion_sdk/calvin_files/fusion/src/FusionBPMAPData.cpp\
>> fusion_sdk/calvin_files/fusion/src/FusionCDFData.cpp\
>> fusion_sdk/calvin_files/fusion/src/FusionCDFQCProbeSetNames.cpp\
>> fusion_sdk/calvin_files/fusion/src/FusionCELData.cpp\
>> fusion_sdk/calvin_files/fusion/src/GCOSAdapter/
>> GCOSCELDataAdapter.cpp\
>> fusion_sdk/calvin_files/parameter/src/ParameterNameValueType.cpp\
>> fusion_sdk/calvin_files/parsers/src/CDFFileReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/CelFileReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/DataGroupHeaderReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/DataGroupReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/DataSetHeaderReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/DataSetReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/FileHeaderReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/FileInput.cpp\
>> fusion_sdk/calvin_files/parsers/src/GenericDataHeaderReader.cpp\
>> fusion_sdk/calvin_files/parsers/src/GenericFileReader.cpp\
>> fusion_sdk/calvin_files/utils/src/AffymetrixGuid.cpp\
>> fusion_sdk/calvin_files/utils/src/DateTime.cpp\
>> fusion_sdk/calvin_files/utils/src/FileUtils.cpp\
>> fusion_sdk/calvin_files/utils/src/StringUtils.cpp\
>> fusion_sdk/calvin_files/utils/src/checksum.cpp\
>> fusion_sdk/file/BPMAPFileData.cpp\
>> fusion_sdk/file/BPMAPFileWriter.cpp\
>> fusion_sdk/file/CDFFileData.cpp\
>> fusion_sdk/file/CELFileData.cpp\
>> fusion_sdk/file/FileIO.cpp\
>> fusion_sdk/file/FileWriter.cpp\
>> R_affx_cel_parser.cpp\
>> R_affx_cdf_parser.cpp\
>> R_affx_cdf_extras.cpp\
>> R_affx_bpmap_parser.cpp
>>
>> OBJS=$(PKG_SOURCES:.cpp=.o)
>>
>> all: $(SHLIB)
>
> --
> Brian D. Ripley, ripley_at_stats.ox.ac.uk
> Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
> University of Oxford, Tel: +44 1865 272861 (self)
> 1 South Parks Road, +44 1865 272866 (PA)
> Oxford OX1 3TG, UK Fax: +44 1865 272595



R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Wed 11 Jul 2007 - 18:47:30 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Thu 12 Jul 2007 - 21:36:23 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.