Re: [R] strange behaviour of memory management

From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk>
Date: Thu 20 Oct 2005 - 18:51:46 EST

On Thu, 20 Oct 2005, Meinhard Ploner wrote:

> Hi all!
> My system: R 2.1.1, Mac OS X 10.4.2.
>
> I have a very memory-consuming job for R, consisting of a function
> calling some other functions, working often with matrices of size
> 100.000 x 300. If I call the job directly after starting R the job
> takes overall 40min, however only 7min of process time. I assume the
> large difference is through memory-handling which doesn't count as
> process time. If i start the job after I make some shorter runs, some
> programming, then the job stops by reaching a memory limit. It seems
> that R doesn't release all the memory even if it don't adds global
> objects.

What is the message? Most often this happens not because memory is not available but because contiguous memory is not available. You have only 3Gb (or less) of process address space, and that can get fragmented enough not to find holes for objects of about 240Mb each (assuming numerical matrices).

> Further I'm interesited if for a UNIX-derivate like Mac OS X gc() or
> rm(localObjects) (used in local functions) make any difference/
> advantage??

gc() unlikely (R has probably already tried that, but we haven't seen the message). rm(localObjects): yes it can help, even for small objects.

If my guess is right, the real answer is a 64-bit OS.

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Received on Thu Oct 20 18:56:30 2005

This archive was generated by hypermail 2.1.8 : Sun 23 Oct 2005 - 19:07:30 EST