RE: R-alpha: Memory Requirements for Large Datasets

Matthew R. Nelson (mrn@superh.hg.med.umich.edu)
Tue, 4 Mar 1997 09:20:42 -0500 (EST)


Date: Tue, 4 Mar 1997 09:20:42 -0500 (EST)
From: "Matthew R. Nelson" <mrn@superh.hg.med.umich.edu>
Subject: RE: R-alpha: Memory Requirements for Large Datasets
To: Martyn Plummer <plummer@iarc.fr>
In-Reply-To: <XFMail.970304134844.plummer@iarc.fr>
Message-Id: <Pine.3.87.9703040942.B15912-0100000@superh.hg.med.umich.edu>


Here at work, I am working on a Linux-ELF (i586-90) with 40 MB RAM.  Even 
cranking up my startup arguements to
     R -v 30 -n 10000000
the largest dataset that I could read in was 53 x 2000 (twice the size of 
the previous largest with the default startup values), which corresponded 
to an ASCII file of 635 KB.

I suppose I could attempt to break the file up and read it in piece by 
piece, then bind them together.

Thanks for your replies.

Matt


On Tue, 4 Mar 1997, Martyn Plummer wrote:

> 
> On 04-Mar-97 mrn@superh.hg.med.umich.edu wrote:
> 
> >I have been frustrated by the apparently memmory hungry nature of R.  I 
> >have attempted to read in a matrix (read.table) that is a little over 1MB 
> >in size (~4000 observations with 43 traits), but am told I lack 
> >sufficient memmory.  It is not until I pare this dataset down to 1000 
> >observations (~0.25 MB) that R will accept it.  Running R on Linux-ELF 
> >with 16 MB of memmory (on i586-75), I installed 8 MB additional RAM, but 
> >still find R incapable of retrieving much more.  
> 
> You didn't say whether you tried the -v flag to increase the size of the
> vector heap (default is only 2Mb). This is documented in the man page
> (R.1) which can be found in the top level of the R source directory tree
> if you haven't installed it already.
> 
> You'll find that you need a lot more memory to read a dataset in than you
> do to work with it. 
> 
> By the way if someone wants to explain what a cons cell is (the option set
> with the -n flag) I'd be very grateful. I have found that I also need to 
> increase this from the default value when reading large data sets.
> 
> Martyn.

----------------------------------------------------------------------------
Matthew R. Nelson
Dept. of Human Genetics
University of Michigan              http://www-personal.umich.edu/~ticul/
4711 Medical Science II             email: ticul@umich.edu
Ann Arbor, MI  48109-0618           phone: (313) 647-3151
  "The many truths we cling to depend upon our own point of view"
            ------              - Obi-Wan Kanobi, Return of the Jedi
----------------------------------------------------------------------------


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
r-testers mailing list -- For info or help, send "info" or "help",
To [un]subscribe, send "[un]subscribe"
(in the "body", not the subject !)  To: r-testers-request@stat.math.ethz.ch
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-