[R] Memory allocation problem with large dataset

From: Pamela Allen <allen_at_zoology.ubc.ca>
Date: Mon, 11 Feb 2008 14:44:10 -0800

Hello All,

I have a problem when I try and run an nlme model with an added correlation structure on a large dataset. This is not surprising, but I am not sure how to fix this problem. I am using R 2.6.1, and I have had similar problems in S-plus.

My dataset is mass growth data from the same 8 animals over 10 years, and is a data.frame with 23,598 rows and 5 columns, and using object.size(data)= 664448 bytes. My first model, using the VonBertalanffy growth function:

vanfemfunc<-function(age.years, A, lk, t0) A*((1-exp(-exp(lk)*(age.years-t0)))^3)

vanfemfit1<-nlme(model=mass~vanfemfunc(age.years, A, lk, t0), data=vanfem, fixed=A+lk+t0~1, start=c("A"=201.8, "lk"=-1.17, "t0"=-2.71))

This fits fine, and has an object size of 3605660 bytes. When I try this second model that involves a correlation structure for the random effects:

vanfemfit2<-update(vanfemfit1, correlation=corAR1())

I receive the following error message:

Error: cannot allocate vector of size 392.9 Mb In addition: Warning messages:
1: In corFactor.corAR1(object) :
  Reached total allocation of 1022Mb: see help(memory.size)

I have looked through all of the R-help archives, but nothing that was previously suggested has worked so far. I have a 32-bit Windows OS with 1 Gb RAM, and I have also tried all of the following functions on a 64-bit Windows OS with 2 Gb RAM. Here are the functions I have tried:

  1. Using --max-mem-size. I changed the upper limit to 2047, which was the maximum value allowed for both the 32-bit OS and the 64-bit OS. The error message for vanfemfit2 now reads "Error: cannot allocate vector of size 392.9 Mb".
  2. Using the function memory.limit(size=4095). If I increase it any higher for both the 32-bit and 62-bit OS, I get the error message "Don't be silly!: your machine has a 4 Gb address limit". Now when I try the model, the error message says "Error: cannot allocate vector of size 392.9 Mb" . Calling memory.size(max=T) produces 1126.44 Mb for the 32-bit OS, but 438.38 for the 64-bit OS. Calling memory.size(max=F) after I try vanfemfit2 produces 1038.02 Mb for the 32-bit, and 417.90 for the 64-bit. If I use gc() and then call memory.size(max=F) again, the output is 240.90 for the 32-bit, and 13.69 for the 64-bit. If I use gc() and then try vanfemfit2 again, I get the same error message on both computers.

I cannot split up the data into smaller pieces to make this function work, as I need all of the data to produce a proper growth curve, although I have reduced the data to 2,000 to test vanfemfit2, and it did work. Any advice would be much appreciated. Thank you!

-Pam Allen
MSc Candidate

R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Mon 11 Feb 2008 - 22:54:52 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Tue 12 Feb 2008 - 00:30:14 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive