From: Patrick Giraudoux <patrick.giraudoux_at_univ-fcomte.fr>

Date: Fri 06 Jan 2006 - 09:02:07 EST

R-help@stat.math.ethz.ch mailing list

https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Fri Jan 06 09:09:24 2006

Date: Fri 06 Jan 2006 - 09:02:07 EST

Thanks a lot Jim. Forwarded to Aaron (who rised the question) and the R-help list...

Patrick

jim holtman a écrit :

*> The size matrix you are allocating (18227 x 18227) would require 2.6GB
**> of memory and that is what the error message is saying that you only
**> have 0.5GB (512MB) available. You also need about 3-4x the largest
**> object so that you can do any calculations on it due to extra copies
**> that might be made of the data.
**>
**> Now your 2000 x 2000 will require about 32MB of memory that that is
**> why it is comfortable in running on your system with 512MB.
**>
**> If you are running on a Windows machine, you may want to use the
**> --max-memory command line parameter to set the memory size. I have
**> 1GB and set the max to 800MB. You may also want to remove any extra
**> objects and do 'gc()' before working with a large object to try and
**> get memory cleared up.
**>
**>
**> On 1/5/06, *Patrick Giraudoux* <patrick.giraudoux@univ-fcomte.fr
*

> <mailto:patrick.giraudoux@univ-fcomte.fr>> wrote:

*>
**> Dear Aaron,
**>
**> I am really a tool user and not a tool maker (actually an ecologist
**> doing some biostatistics)... so, I take the liberty of sending a
**> copy of
**> this e-mail to the r-help list where capable computer persons and true
**> statisticians may provide more relevant information and also to Paulo
**> Ribeiro and Peter Diggle, the authors of geoR..
**>
**> I really feel that your huge matrix cannot be handled in R that easy,
**> and I get the same kind of error as you:
**>
**> > m=matrix(1,ncol=18227, nrow=18227)
**> Error: cannot allocate vector of size 2595496 Kb
**> In addition: Warning messages:
**> 1: Reached total allocation of 511Mb: see help( memory.size)
**> 2: Reached total allocation of 511Mb: see help(memory.size)
**>
**> However, if you want to compute a distance matrix, have a look to the
**> function:
**>
**> ?dist
**>
**> and try it... You will not have to create the distance matrix
**> yourself
**> from the coordinates file (but you may meet meory problem anyway).
**> Still
**> more straigthfully, if you intend to use interpolation methods such as
**> kriging, you don't need to manage the distance matrix building by
**> yourself. See:
**>
**> library(geoR)
**> library(help=geoR)
**> ?variog
**> ?variofit
**> ?likfit?
**>
**> etc...
**>
**> If you want further use geostatics (eg via geoR or gstat), you will
**> anyway have to manage with memory limits (not due to R). On my
**> computer
**> (portable HP compaq nx7000) I can hardly manage with more than 2000
**> observations using geoR (far from your 18227 observations). I had to
**> krige on a dataset with 9000 observations recently and has been led to
**> subsample randomly 2000 values. You can however try and increase the
**> memory allocated to R on your computer. The size limit is hardware
**> dependent, eg:
**>
**> ?memory.size
**> ?memory.limit
**> / memory.limit(size=500000000) /
**>
**> Another way may be to perform local kriging (eg kriging within the
**> dataset on a fixed radius), but to my knowldege this cannot be
**> done with
**> geoR (unfortunately). The library gstat offers this option, but
**> has much
**> more limited possibilities than geoR considering other issues
**> (variogram
**> analysis, etc...).
**>
**> library(gstat)
**> ?krige
**>
**> see argument 'maxdist'
**>
**> Hope this can help,
**>
**> Kind regards
**>
**>
**> Patrick Giraudoux
**> --
**>
**> Department of Environmental Biology
**> EA3184 usc INRA
**> University of Franche-Comte
**> 25030 Besancon Cedex
**> (France)
**>
**> tel. +33 381 665 745
**> fax +33 381 665 797
**> http://lbe.univ-fcomte.fr
**>
**>
**> Aaron Swoboda a écrit :
**>
**> > Dear Sir:
**> >
**> >
**> > I ran across your post to the R-help archive from February 9it is
**> > attached below since it was nearly two years ago!). I am
**> beginning to
**> > learn R and am interested in analyzing some of my data in a spatial
**> > context. I am having a problem that seems similar to the problem you
**> > encountered, trying to work with a large matrix in Windows XP).
**> I am
**> > wondering if you could help steer me in the direction that
**> helped you
**> > solve your problem. I am trying to construct a distance matrix
**> > containing the all of the distances between my 18,000 observations.
**> > Trying to make a matrix that large gets an error message...
**> >
**> > m=matrix(1,ncol=18227, nrow=18227)
**> >
**> > Error: cannot allocate vector of size 2595496 Kb
**> > In addition: Warning messages:
**> > 1: Reached total allocation of 1023Mb: see help(memory.size)
**> >
**> > Thank you for any help you may be able to send my way.
**> >
**> > ~Aaron Swoboda
**> >
**> > Below is your posting to the R-help list...
**> >
**> >
**> > Dear all,
**> >
**> > I a read with great interest the e-mails related to Arnav Sheth
**> about
**> > memory limitation when computing a distance matrix. I suspect
**> > that I will also meet some memory limitation using GeoR. I am
**> > currently running GeoR on a geodata object including 2686
**> geographical
**> > coordinates.
**> >
**> > krige.conv() can handle it (it takes 10-15 mn of computing) but
**> > requests an increased memory.
**> >
**> > /> memory.limit(size=500000000) /
**> >
**> > When the computing is completed, the computer speed is considerably
**> > slowed down for any application. It is thus most necessary to
**> > shut it down and restart. I will probably have to handle a set of
**> > 5000-6000 coordinates in once in the next few months. I wonder if
**> > it will go through it on my plateform (Windows XP and compaq
**> nx7000).
**> > If not, will the limitation due to R or to Windows? Does an
**> > alternate solution exist?
**> >
**> > Thanks for any hint,
**> >
**> > Patrick Giraudoux
**> >
**> >--
**> > Dr. Aaron M. Swoboda
**> > 3208 Posvar Hall
**> > Graduate School of Public and International Affairs
**> > University of Pittsburgh
**> > Pittsburgh, PA 15260
**> >
**> > e-mail: aswoboda@gspia.pitt.edu <mailto:aswoboda@gspia.pitt.edu>
**> > Office: 412-648-7604
**> > Fax: 412-648-2605
**> >
**> >
**>
**> [[alternative HTML version deleted]]
**>
**>
**>
**> ______________________________________________
**> R-help@stat.math.ethz.ch <mailto:R-help@stat.math.ethz.ch> mailing
**> list
**> https://stat.ethz.ch/mailman/listinfo/r-help
**> PLEASE do read the posting guide!
**> http://www.R-project.org/posting-guide.html
**>
**>
**>
**>
**> --
**> Jim Holtman
**> Cincinnati, OH
**> +1 513 247 0281
**>
**> What the problem you are trying to solve?
*

[[alternative HTML version deleted]]

R-help@stat.math.ethz.ch mailing list

https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Fri Jan 06 09:09:24 2006

*
This archive was generated by hypermail 2.1.8
: Fri 06 Jan 2006 - 14:16:21 EST
*