Re: [R] load ing and saving R objects

From: bogdan romocea <br44114_at_gmail.com>
Date: Wed 15 Jun 2005 - 02:34:34 EST


> On Tue, 14 Jun 2005, Prof Brian Ripley wrote:
> If your file system does not like 15000 files you can always
> save in a DBMS.

Or, switch to a better/more appropriate file system: http://en.wikipedia.org/wiki/Comparison_of_file_systems ReiserFS would allow you to store up to about 1.2 million files in a directory.

> -----Original Message-----
> From: Prof Brian Ripley [mailto:ripley@stats.ox.ac.uk]
> Sent: Tuesday, June 14, 2005 10:41 AM
> To: Barry Rowlingson
> Cc: r-help@stat.math.ethz.ch; Richard Mott
> Subject: Re: [R] load ing and saving R objects
>
>
> On Tue, 14 Jun 2005, Barry Rowlingson wrote:
>
> > Richard Mott wrote:
> >> Does anyone know a way to do the following:
> >>
> >> Save a large number of R objects to a file (like load()
> does) but then
> >> read back only a small named subset of them . As far as I can see,
> >> load() reads back everything.
> >
> > Save them to individual files when you generate them?
> >
> > for(i in 1:15000){
> >
> > m=generateBigMatrix(i)
> >
> > filename=paste("BigMatrix-",i,".Rdata",sep='')
> > save(m,file=filename)
> > }
> >
> > Note that load will always overwrite 'm', so to load a
> sample of them in
> > you'll need to do something like this:
> >
> > bigSamples=list()
> >
> > for(i in sample(15000,N)){
> > filename=paste("BigMatrix-",i,".Rdata",sep='')
> > load(filename)
> > bigSamples[[i]]=m
> > }
> >
> > But there may be a more efficient way to string up a big list like
> > that, I can never remember - get it working, then worry
> about optimisation.
>
> (Yes, use bigSamples <- vector("list", 15000) first.)
>
> > I hope your filesystem is happy with 15000 objects in it. I would
> > dedicate a folder or directory for just these objects'
> files, since it
> > then becomes near impossible to see anything other than the
> big matrix
> > files...
>
> .readRDS/.saveRDS might be a better way to do this, and avoids always
> restoring to "m".
>
> If your file system does not like 15000 files you can always
> save in a
> DBMS.
>
> I did once look into restoring just some of the objects in a save()ed
> file, but it is not really possible to do so efficiently due
> to sharing
> between objects.
>
> --
> Brian D. Ripley, ripley@stats.ox.ac.uk
> Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
> University of Oxford, Tel: +44 1865 272861 (self)
> 1 South Parks Road, +44 1865 272866 (PA)
> Oxford OX1 3TG, UK Fax: +44 1865 272595
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Wed Jun 15 02:38:29 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:32:36 EST