Re: [R] large dataset!

From: Andrew Robinson <A.Robinson_at_ms.unimelb.edu.au>
Date: Mon 03 Jul 2006 - 07:05:35 EST

Jennifer,

it sounds like that's too much data for R to hold in your computer's RAM. You should give serious consideration as to whether you need all those data for the models that you're fitting, and if so, whether you need to do them all at once. If not, think about pre-processing steps, using e.g. SQL command, to pull out the data that you need. For example, if the data are spatial, then think about analyzing them by patches.

Good luck,

Andrew

On Sun, Jul 02, 2006 at 10:12:25AM -0400, JENNIFER HILL wrote:
>
> Hi, I need to analyze data that has 3.5 million observations and
> about 60 variables and I was planning on using R to do this but
> I can't even seem to read in the data. It just freezes and ties
> up the whole system -- and this is on a Linux box purchased about
> 6 months ago on a dual-processor PC that was pretty much the top
> of the line. I've tried expanding R the memory limits but it
> doesn't help. I'll be hugely disappointed if I can't use R b/c
> I need to do build tailor-made models (multilevel and other
> complexities). My fall-back is the SPlus big data package but
> I'd rather avoid if anyone can provide a solution....
>
> Thanks!!!!
>
> Jennifer Hill
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html

-- 
Andrew Robinson  
Department of Mathematics and Statistics            Tel: +61-3-8344-9763
University of Melbourne, VIC 3010 Australia         Fax: +61-3-8344-4599
Email: a.robinson_at_ms.unimelb.edu.au         http://www.ms.unimelb.edu.au

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Received on Mon Jul 03 07:10:06 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Mon 03 Jul 2006 - 14:14:36 EST.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.