Re: [R] assigning and saving datasets in a loop, with names changing with "i"

From: Greg Snow <Greg.Snow_at_imail.org>
Date: Thu, 20 Dec 2007 13:57:56 -0700

Have you looked at the SQLiteDF package? It seems like it would do what you want in a better way and much simpler. Even if that does not work then a database approach (look at the other db packages, probably RODBC first) could be simpler, faster, and easier. You may also want to look at the g.data package for another approach.

Depending on what you are doing you may also want to look at the biglm package (some similar functionality is in SQLiteDF).

Hope this helps,

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow_at_imail.org
(801) 408-8111
 
 


> -----Original Message-----
> From: r-help-bounces_at_r-project.org
> [mailto:r-help-bounces_at_r-project.org] On Behalf Of Marie
> Pierre Sylvestre
> Sent: Tuesday, December 18, 2007 7:25 PM
> To: r-help_at_r-project.org
> Subject: [R] assigning and saving datasets in a loop,with
> names changing with "i"
>
> Dear R users,
>
> I am analysing a very large data set and I need to perform
> several data manipulations. The dataset is so big that the
> only way I can play with it without having memory problems
> (E.g. "cannot allocate vectors of size...") is to write a
> batch script to:
>
> 1. cut the data into pieces
> 2. save the pieces in seperate .RData files 3. Remove
> everything from the environment 4. load one of the piece 5.
> perform the manipulations on it 6. save it and remove it from
> the environment 7. Redo 4-6 for every piece 8. Merge
> everything together at the end
>
> It works if coded line by line but since I'll have to perform
> these tasks on other data sets, I am trying to automate this
> as much as I can.

>
> I am using a loop in which I used 'assign' and 'get' (pseudo
> code below).
> My problem is when I use 'get', it prints the whole object on
> the screen.
> I am wondering whether there is a more efficient way to do
> what I need to do. Any help would be appreciated. Please keep
> in mind that the whole process is quite computer-intensive,
> so I can't keep everything in the environment while R
> performs calculations.
>
> Say I have 1 big dataframe called data. I use 'split' to
> divide it into a list of 12 dataframes (call this list my.list)
>
> my.fun is a function that takes a dataframe, performs several
> manipulations on it and returns a dataframe.
>
>
> for (i in 1:12){
> assign( paste( "data", i, sep=""), my.fun(my.list[i])) #
> this works
> # now I need to save this new object as a RData.
>
> # The following line does not work
> save(paste("data", i, sep = ""), file = paste(
> paste("data", i, sep = ""), "RData", sep=".")) }
>
> # This works but it is a bit convoluted!!!
> temp <- get(paste("data", i, sep = ""))
> save(temp, file = "lala.RData")
> }
>
>
> I am *sure* there is something more clever to do but I can't
> find it. Any help would be appreciated.
>
> best regards,
>
> MP
>
> ______________________________________________
> R-help_at_r-project.org mailing list

> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
______________________________________________ R-help_at_r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Received on Thu 20 Dec 2007 - 21:02:14 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Thu 20 Dec 2007 - 21:30:20 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.