Re: [R] memory limits in R loading a dataset and using the package tree

From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk>
Date: Thu 04 Jan 2007 - 23:25:47 GMT

Please read the rw-FAQ Q2.9. There are ways to raise the limit, and you have not told us that you used them (nor the version of R you used, which matters as the limits are version-specific).

Beyond that, there are ways to use read.table more efficiently: see its help page and the 'R Data Import/Export' manual. In particular, did you set nrows and colClasses?

But for the size of problem you have I would use a 64-bit build of R.

On Thu, 4 Jan 2007, domenico pestalozzi wrote:

> I think the question is discussed in other thread, but I don't exactly find
> what I want .
> I'm working in Windows XP with 2GB of memory and a Pentium 4 - 3.00Ghx.
> I have the necessity of working with large dataset, generally from 300,000
> records to 800,000 (according to the project), and about 300 variables
> (...but a dataset with 800,000 records could not be "large" in your
> opinion...). Because of we are deciding if R will be the official software
> in our company, I'd like to say if the possibility of using R with these
> datasets depends only by the characteristics of the "engine" (memory and
> processor).
> In this case we can improve the machine (for example, what memory you
> reccomend?).
>
> For example, I have a dataset of 200,000 records and 211 variables but I
> can't load the dataset because R doesn't work : I control the loading
> procedure (read.table in R) by using the windows task-manager and R is
> blocked when the file paging is 1.10 GB.
> After this I try with a sample of 100,000 records and I can correctly load
> tha dataset, but I'd like to use the package tree, but after some seconds (
> I use this tree(variable1~., myDataset) ) I obtain the message "Reached
> total allocation of 1014Mb".
>
> I'd like your opinion and suggestion, considering that I could improve (in
> memory) my computer.
>
> pestalozzi
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Received on Fri Jan 05 10:30:54 2007

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Fri 05 Jan 2007 - 00:30:25 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.