Re: [R] 64-bit R and cache memory

From: Gad Abraham <gabraham_at_csse.unimelb.edu.au>
Date: Sat, 17 May 2008 21:12:41 +1000

Joram Posma wrote:
> Dear all,
>
> I have a few questions regarding the 64 bit version of R and the cache
> memory R uses.
>
> -----------------------------------
> Computer & software info:
>
> OS: kUbuntu Feasty Fawn 7.04 (64-bit)
> Processor: AMD Opteron 64-bit
> R: version 2.7.0 (64-bit)
> Cache memory: currently 16 GB (was 2 GB)
> Outcome of 'limit' command in shell: cputime unlimited, filesize
> unlimited, datasize unlimited, stacksize 8192 kbytes, coredumpsize 0
> kbytes, memoryuse unlimited, vmemoryuse unlimited, descriptors 1024,
> memorylocked unlimited, maxproc unlimited
> -----------------------------------
>
> a. We have recently upgraded the cache memory from 2 to 16 GB. However,
> we have noticed that somehow R still swaps memory when datasets
> exceeding 2 GB in size are used. An indication that R uses approx. 2 GB
> of cache memory is that sometimes R also kills the session when datasets
> > 2 GB are loaded. How/where can we see how much cache memory R uses
> (since memory.size and memory.limit are only for windows, and to us
> those might be what we need)? Could this be caused by the limit of the
> stack size (we are not exactly sure what the stack size is either) ?
> b. And how can we increase the cache memory used by R to 14 or even 16
> GB (which might be tricky when running other programs, but still)?
>
> So in general: how can we get R to use the full memory capacity of the
> computer?
>

The term "cache memory" is something entirely different to what you're referring to --- you're talking about RAM.

Anyway, under Linux R will take all the RAM it can get, and if you're running a 64-bit OS on a 64-bit CPU then it should definitely be able to use more than 2GB of RAM.

To see the memory usage, use the utility "top" in the console/terminal.

One thing to note: a dataset of 2GB on disk may take much more than 2GB of RAM when loaded into R, due to the overhead of the metadata and the fact that pointers are 64-bit long as well.

-- 
Gad Abraham
Dept. CSSE and NICTA
The University of Melbourne
Parkville 3010, Victoria, Australia
email: gabraham_at_csse.unimelb.edu.au
web: http://www.csse.unimelb.edu.au/~gabraham

______________________________________________
R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Received on Sat 17 May 2008 - 11:19:31 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Sat 17 May 2008 - 18:30:38 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive