Re: [R] memory

From: Sean O'Riordain <sean.oriordain_at_gmail.com>
Date: Tue 30 Aug 2005 - 17:56:52 EST

Ferran,

What are you trying to do with such a large matrix? with 7e9 cells and a linear algorithm which is quite unlikely, your problem solution is likely to take a "very long time"(tm)... just quickly... at one micro-second per operation (very optimistic?) and 7e9 operations, thats
> 7e9/1e6/60

[1] 116.6667
minutes...

if we're doing something a little more complicated than linear, say O(n^2.5) on a square matrix of 7e9 cells, then we're talking
> (7e9^.5)^2.5/1e6/60

[1] 33745.92
minutes...

As Brian Ripley said, if you really want to to this then you must use another operating system which can handle more than 32-bit addressing, one such would be linux running and built for a 64-bit platform - of which there are a few.

cheers!
Sean

On 30/08/05, Ferran Carrascosa <ferran.carrascosa@gmail.com> wrote:
> Thanks Prof Brian for your answers,
> I have read about 'ref' package to work with more efficient memory
> work. Anybody know if this package could help me to work with a
> 700.000 x 10.000 matrix?
>
> I will have problems with ref package on:
> - Limit of 2 Gb in R for Windows.
> -The maximum cells in one object 2*10^9 (aprox.)
>
> Thanks in advance,
> --
> Ferran Carrascosa
>
>
> 2005/8/30, Prof Brian Ripley <ripley@stats.ox.ac.uk>:
> > On Mon, 29 Aug 2005, Ferran Carrascosa wrote:
> >
> > > Hi,
> > >
> > > I have a matrix with 700.000 x 10.000 cells with floating point data.
> > > I would like to work with the entire table but I have a lot of memory
> > > problems. I have read the ?memory
> > > I work with Win 2000 with R2.1.0
> > >
> > > The only solution that I have applied is:
> > >> memory.limit(size=2048)
> > >
> > > But now my problems are:
> > > - I need to work with more than 2 Gb. How I can exceed this limit?
> >
> > Re-read the rw-FAQ, or (preferably) get a more capable OS on a 64-bit CPU.
> >
> > > - When apply some algorithms, the maximum cells in one object 2*10^9
> > > (aprox.) is reached.
> >
> > You will never get that many cells (that is the address space in bytes,
> > and they are several bytes each). Please do as the posting guide asks
> > and report accurately what happened.
> >
> > > Please could you send me some advises/strategies about the work with
> > > large amount of data in R?
> > >
> > > R have a way to work with less memory needs?
> >
> > Your matrix has 7e09 cells (assuming you are using . as a thousands
> > separator) and needs 5.6e10 bytes to store. Your OS has a memory address
> > limit of 3.2e09 bytes. Don't blame R for being limited by your OS.
> >
> > --
> > Brian D. Ripley, ripley@stats.ox.ac.uk
> > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
> > University of Oxford, Tel: +44 1865 272861 (self)
> > 1 South Parks Road, +44 1865 272866 (PA)
> > Oxford OX1 3TG, UK Fax: +44 1865 272595
> >
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Tue Aug 30 18:04:37 2005

This archive was generated by hypermail 2.1.8 : Sun 23 Oct 2005 - 16:07:07 EST