Re: [R] Reading large files in R

From: Berton Gunter <gunter.berton_at_gene.com>
Date: Tue 09 Aug 2005 - 05:35:52 EST


... and it is likely that even if you did have enough memory (several times the size of the data are generally needed) it would take a very long time.

If you do have enough memory and the data are all of one type -- numeric here -- you're better off treating it as a matrix rather than converting it to a data frame.

"The business of the statistician is to catalyze the scientific learning process." - George E. P. Box    

> -----Original Message-----
> From: r-help-bounces@stat.math.ethz.ch
> [mailto:r-help-bounces@stat.math.ethz.ch] On Behalf Of
> Adaikalavan Ramasamy
> Sent: Monday, August 08, 2005 12:02 PM
> To: Jean-Pierre Gattuso
> Cc: r-help@stat.math.ethz.ch
> Subject: Re: [R] Reading large files in R
>
> >From Note section of help("read.delim") :
>
> 'read.table' is not the right tool for reading large matrices,
> especially those with many columns: it is designed to read _data
> frames_ which may have columns of very different classes. Use
> 'scan' instead.
>
> So I am not sure why you used 'scan', then converted it to a
> data frame.
>
> 1) Can provide an sample of the data that you are trying to read in.
> 2) How much memory does your machine has ?
> 3) Try reading in the first few lines using the nmax argument in scan.
>
> Regards, Adai
>
>
>
> On Mon, 2005-08-08 at 12:50 -0600, Jean-Pierre Gattuso wrote:
> > Dear R-listers:
> >
> > I am trying to work with a big (262 Mb) file but apparently
> reach a
> > memory limit using R on a MacOSX as well as on a unix machine.
> >
> > This is the script:
> >
> > > type=list(a=0,b=0,c=0)
> > > tmp <- scan(file="coastal_gebco_sandS_blend.txt", what=type,
> > sep="\t", quote="\"", dec=".", skip=1, na.strings="-99",
> nmax=13669628)
> > Read 13669627 records
> > > gebco <- data.frame(tmp)
> > Error: cannot allocate vector of size 106793 Kb
> >
> >
> > Even tmp does not seem right:
> >
> > > summary(tmp)
> > Error: recursive default argument reference
> >
> >
> > Do you have any suggestion?
> >
> > Thanks,
> > Jean-Pierre Gattuso
> >
> > ______________________________________________
> > R-help@stat.math.ethz.ch mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> >
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Tue Aug 09 05:44:58 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:39:45 EST