[R] Re : growing a list sequentially -- memory management

From: justin bem <justin_bem_at_yahoo.fr>
Date: Fri, 14 Nov 2008 15:36:08 +0000 (GMT)

you may try something like  

myres <-tapply(1:n, as.factor(1:n) gen)

but n in very large in your case.

you may create an external database and fill it

connect to you database

library(RODC)
con<-odbcConnect(...)

st<-seq(1,n, by=h) 
# since n=157347, h can be set to 1573 to have around that 100 loops

for ( i in 1: length(sh)){

    index<-st(i):st(i+1)
    myres(tapply(index, factor(index),gen
    sqlwrite(myres,connexion,append=(i>1))
}

It will be more faster .
 Justin BEM
BP 1917 Yaoundé
Tél (237) 99597295
(237) 22040246


De : Patrick Burns <pburns_at_pburns.seanet.com>
À : erwann rogard <erwann.rogard_at_gmail.com>
Cc : r-help_at_r-project.org

Envoyé le : Vendredi, 14 Novembre 2008, 12h37mn 35s Objet : Re: [R] growing a list sequentially -- memory management

The recommended technique is to create objects as their final size and then subscript into them with your data.

My intuition (which is often brutally wrong) tells me that your case should not be overly traumatic. So I'm suspicious that you are fragmenting memory in other ways as well.

Patrick Burns
patrick_at_burns-stat.com
+44 (0)20 8525 0696
http://www.burns-stat.com
(home of S Poetry and "A Guide for the Unwilling S User")

erwann rogard wrote:
> hello,
>
> i have something like:
>
> out<-list()
>
> for(i in 1:n){
>  data<-gen(...) #fixed size data
>  out[[i]]<- fun(data)
> }
>

>> object.size(out[[1]])
>>   
> 6824
>
> In principle 1 GB should allow
>
> n = 1024^3/6824 = 157347?
>
> i have about 2GB are not taken by other processes. however, I can see the
> memory shrinking quite rapidly on my system monitor and have to stop the
> simulation after only n=300. why such a discrepancy? any remedy?
>
> x86_64-pc-linux/RKWard/R2.8.0/ 4GB
>
> thanks.
>
>     [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help_at_r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
>



R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
      
	[[alternative HTML version deleted]]


______________________________________________

R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Fri 14 Nov 2008 - 15:45:27 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Fri 14 Nov 2008 - 16:30:25 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive