Re: [Rd] .Call and to reclaim the memory by allocVector

From: Yongchao Ge <Yongchao.Ge_at_mssm.edu>
Date: Tue, 28 Aug 2007 13:02:16 -0400 (EDT)

Hi Seth,

Thank you for the suggestion. Because of using .Call (which does not copy the value) for both parts of my program, there is no extra copy shown by tracemem(). Anyway, the information shown by gc() is very misleading as stated by Prof. Ripley, especially after creating and removing a couple of large R datasets and applying the function gc() a couple of times.

As shown by "ps aux", there is no "memory leak" from .Call. It's a big relief to me. Mysteriously, my program works now for storing the intermediate results as a 660M R object. I can run the same function as often as I want. The maximum space taken by the program has never exceeded 1.8G as I expected. The disappearance of taking too much memory from .Call may be due to a recompile of my C code or a restart of the linux or a fresh mind after the weekend.

Thank you and Prof. Ripley for the suggestions. It helps me to stay focused.

Yongchao

On Sat, 25 Aug 2007, Seth Falcon wrote:

> Hi Yongchao,
>
> Yongchao Ge <Yongchao.Ge_at_mssm.edu> writes:
>> Why am I storing a large dataset in the R? My program consist of two
>> parts. The first part is to get the intermediate results, the computation
>> of which takes a lot of time. The second part contains many
>> different functions to manipulate the the intermediate
>> results.
>>
>> My current solution is to save intermediate result in a temporary file,
>> but my final goal is to to save it as an R object. The "memory leak" in
>> .Call stops me from doing this and I'd like to know if I can have a clean
>> solution for the R package I am writing.
>
> There are many examples of packages that use .Call to create large
> objects. I don't think there is a "memory leak".
>
> One thing that may be catching you up is that because of R's
> pass-by-value semantics, you may be ending up with multiple copies of
> the object on the R side during some of your operations. I would
> recommend recompiling with --enable-memory-profiling and using
> tracemem() to see if you can identify places where copies of your
> large object are occurring. You can also take a look at
> Rprof(memory.profile=TRUE).
>
> + seth
>
>
> --
> Seth Falcon | Computational Biology | Fred Hutchinson Cancer Research Center
> BioC: http://bioconductor.org/
> Blog: http://userprimary.net/user/



R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Wed 29 Aug 2007 - 07:05:44 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Wed 29 Aug 2007 - 19:39:43 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.