Re: R-alpha: Memory Requirements for Large Datasets

Peter Dalgaard BSA (p.dalgaard@kubism.ku.dk)
04 Mar 1997 16:31:08 +0100


To: "Matthew R. Nelson" <mrn@superh.hg.med.umich.edu>
Subject: Re: R-alpha: Memory Requirements for Large Datasets
From: Peter Dalgaard BSA <p.dalgaard@kubism.ku.dk>
Date: 04 Mar 1997 16:31:08 +0100
In-Reply-To: "Matthew R. Nelson"'s message of Tue, 4 Mar 1997 09:20:42 -0500 (EST)
Message-Id: <x29143d683.fsf@bush.kubism.ku.dk>

"Matthew R. Nelson" <mrn@superh.hg.med.umich.edu> writes:

> 
> 
> Here at work, I am working on a Linux-ELF (i586-90) with 40 MB RAM.  Even 
> cranking up my startup arguements to
>      R -v 30 -n 10000000

Er, am I missing something? I thought that a cons cell was the same as
a type SEXPREC object (cf. include/Defn.h, 16 bytes in size, so that
that call reserves ~160 MB of them. That could give problems all on
its own...

> the largest dataset that I could read in was 53 x 2000 (twice the size of 
> the previous largest with the default startup values), which corresponded 
> to an ASCII file of 635 KB.
> 
> I suppose I could attempt to break the file up and read it in piece by 
> piece, then bind them together.

- or write a short dyn.load-able C function to do the work. This *is*
a bit silly! Does anyone have ideas on specifications for an efficient
variant of read.table()?

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard@biostat.ku.dk)             FAX: (+45) 35327907
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
r-testers mailing list -- For info or help, send "info" or "help",
To [un]subscribe, send "[un]subscribe"
(in the "body", not the subject !)  To: r-testers-request@stat.math.ethz.ch
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-