Re: [R] memory

From: Andres Legarra <alegarra_at_neiker.net>
Date: Tue 30 Aug 2005 - 17:49:34 EST

Re: [R] memoryFrankly:
you probably CAN NOT unles you have a very, very powerful machine. 5.6e10 bytes = (if I got it right) 35 Gb of memory. Do you have that much?

If you have sparse matrices (those with many, many zeros), perhaps you can work with sparse matrices methods... but if you had them you would probably know that

Andres

--
Andres Legarra Albizu
NEIKER
Apdo. 46 
Vitoria-Gasteiz 01080 Spain
--

----- Original Message ----- 
From: Ferran Carrascosa 
To: Prof Brian Ripley 
Cc: r-help@stat.math.ethz.ch 
Sent: Tuesday, August 30, 2005 9:39 AM
Subject: Re: [R] memory


Thanks Prof Brian for your answers, 
I have read about 'ref' package to work with more efficient memory 
work. Anybody know if this package could help me to work with a 
700.000 x 10.000 matrix? 

I will have problems with ref package on: 
- Limit of 2 Gb in R for Windows. 
-The maximum cells in one object 2*10^9 (aprox.) 

Thanks in advance, 
-- 
Ferran Carrascosa 



2005/8/30, Prof Brian Ripley <ripley@stats.ox.ac.uk>: 

> On Mon, 29 Aug 2005, Ferran Carrascosa wrote:
>
> > Hi,
> >
> > I have a matrix with 700.000 x 10.000 cells with floating point data.
> > I would like to work with the entire table but I have a lot of memory
> > problems. I have read the ?memory
> > I work with Win 2000 with R2.1.0
> >
> > The only solution that I have applied is:
> >> memory.limit(size=2048)
> >
> > But now my problems are:
> > - I need to work with more than 2 Gb. How I can exceed this limit?
>
> Re-read the rw-FAQ, or (preferably) get a more capable OS on a 64-bit CPU.
>
> > - When apply some algorithms, the maximum cells in one object 2*10^9
> > (aprox.) is reached.
>
> You will never get that many cells (that is the address space in bytes,
> and they are several bytes each). Please do as the posting guide asks
> and report accurately what happened.
>
> > Please could you send me some advises/strategies about the work with
> > large amount of data in R?
> >
> > R have a way to work with less memory needs?
>
> Your matrix has 7e09 cells (assuming you are using . as a thousands
> separator) and needs 5.6e10 bytes to store. Your OS has a memory address
> limit of 3.2e09 bytes. Don't blame R for being limited by your OS.
>
> --
> Brian D. Ripley, ripley@stats.ox.ac.uk
> Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
> University of Oxford, Tel: +44 1865 272861 (self)
> 1 South Parks Road, +44 1865 272866 (PA)
> Oxford OX1 3TG, UK Fax: +44 1865 272595
>
______________________________________________ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html [[alternative HTML version deleted]] ______________________________________________ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Received on Tue Aug 30 18:03:46 2005

This archive was generated by hypermail 2.1.8 : Sun 23 Oct 2005 - 16:06:46 EST