Re: [R] rbind wastes memory

From: Duncan Murdoch <murdoch_at_stats.uwo.ca>
Date: Tue 31 May 2005 - 00:08:01 EST

lutz.thieme@amd.com wrote:
> Hello everybody,
>
> if I try to (r)bind a number of large dataframes I run out of memory because R
> wastes memory and seems to "forget" to release memory.
>
> For example I have 10 files. Each file contains a large dataframe "ds" (3500 cols
> by 800 rows) which needs ~20 MB RAM if it is loaded as the only object.
> Now I try to bind all data frames to a large one and need more than 1165MB (!)
> RAM (To simplify the R code, I use the same file ten times):
>
> ________ start example 1 __________
> load(myFile)
> ds.tmp <- ds
> for (Cycle in 1:10) {
> ds.tmp <- rbind(ds.tmp, ds)
> }
> ________ end example 1 __________
>
>
>
> Stepping into details I found the following (comment shows RAM usage after this line
> was executed):
> load(myFile) # 40MB (19MB for R itself)
> ds.tmp <- ds # 40MB; => only a pointer seems to be copied
> x<-rbind(ds.tmp, ds) # 198MB
> x<-rbind(ds.tmp, ds) # 233MB; the same instruction a second time leads to
> # 35MB more RAM usage - why?

I'm guessing your problem is fragmented memory. You are creating big objects, then making them bigger. This means R needs to go looking for large allocations for the replacements, but they won't fit in the spots left by the things you've deleted, so those are being left empty.

A solution to this is to use two passes: first figure out how much space you need, then allocate it and fill it. E.g.

for (Cycle in 1:10) {

     rows[Cycle] <- .... some calculation based on the data ... }

ds.tmp <- data.frame(x=double(sum(rows)), y=double(sum(rows)), ...

for (Cycle in 1:10) {

     ds.tmp[ appropriate rows, ] <- new data }

Duncan Murdoch



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Tue May 31 00:13:44 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:32:16 EST