Re: [R] Memory problems with large dataset in rpart

From: Prof Brian Ripley <>
Date: Tue 18 Oct 2005 - 16:50:39 EST

Looks like you have missed the section in the rw-FAQ entitled

     2.9 There seems to be a limit on the memory it uses!

and not set --max-mem-size (which defaults to 1Gb on your system).

However, it looks like your problem is memory fragmentation, and trying to run 1Gb tasks in a 2Gb address space is intrinsically a problem to which the only solution is a 64-bit version of R.

BTW, /3GB is nothing whatsoever to do with `swap files': if both OS and application are configured correctly it increases the user address space *for that process* to /3GB (whereas swap space is shared between processes).

On Tue, 18 Oct 2005 wrote:

> Dear helpers,
> I am a Dutch student from the Erasmus University. For my Bachelor thesis I
> have written a script in R using boosting by means of classification and
> regression trees. This script uses the function the predefined function
> rpart. My input file consists of about 4000 vectors each having 2210
> dimensions. In the third iteration R complains of a lack of memory,
> although in each iteration every variable is removed from the memory. Thus
> the first two iterations run without any problems.
> My computer runs on Windows XP and has 1 gigabye of internal memory.
> I tried R using more memory by refiguring the swap files as memtioned in
> the FAQ (/3gb), but I didn't succeed in making this work.
> The command round(memory.limit()/1048576.0, 2) gives 1023.48
> If such an increase of memory can not succeed, perhaps the size of the
> rpart object could be reduced by not storing unnecessary information.
> The rpart function call is (the calls of FALSE is to try to reduce the
> size of the fit object):
> fit <- rpart(price ~ ., data = trainingset,
> control=rpart.control(maxdepth=2,cp=0.001),model=FALSE,x=FALSE,y=FALSE)
> This fit object is later called in 2 predict functions, for example:
> predict(fit,newdata=sample)
> Can anybody please help me by letting R use more memory (for example swap)
> or can anybody help me reducing the size of the fit object?

Brian D. Ripley,        
Professor of Applied Statistics,
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________ mailing list
PLEASE do read the posting guide!
Received on Tue Oct 18 16:58:36 2005

This archive was generated by hypermail 2.1.8 : Sun 23 Oct 2005 - 19:01:07 EST