[R] reading big data frame

From: <herodote_at_oreka.com>
Date: Thu 27 Oct 2005 - 00:01:06 EST


hy all,

I'm under linux,
I have now a big file (45 méga bytes), this file is a line of header defining the names of the columns, the rows are only integers and there is 75 columns (and 239096 rows included the header).

Before working with this big file i was working with another one (1.7 méga bytes) , loading it with read.table("data.dat",h=T) (i've tested too with scan("data.dat",what="integer",skip=1)), there where no problems, it where taken approx 1 sec to load it into R.

but now with the big file R it seems that R stay stuck (i've wait more than 5 minutes and R don't come back from the read.table function, i've tested with scan("data.dat",what="integer",skip=1)).

I say it stay stuck because i've spy R with the linux "top" command line, and R begin to take all my memory (no prob with that) and 100% of my cpu (no prob too), then after 3 or 4 minutes it take only 3% of cpu and the memory allocated to R is average 90%, but R never come back from these functions (or i haven't wait enough, and i need to give a result as speed as i can unfortunatly).

I've divided the size of the file by 2 , then it works (i've only tested with scan) but it is too slow...

Are there some solutions to load it in one piece? Or have i to truncate the file and access it piece by piece?

thks all
guillaume.



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Thu Oct 27 00:17:52 2005

This archive was generated by hypermail 2.1.8 : Thu 27 Oct 2005 - 03:28:52 EST