[Rd] R scripts slowing down after repeated called to compiled code

From: Michael Braun <braunm_at_MIT.EDU>
Date: Fri, 25 May 2007 19:12:56 -0400


Thanks in advance to anyone that might be able to help me with this problem. I have not been able to find a reference to it in the documentation on online sources, so I am turning to this group.

I am running R 2.4.1 under Red Hat Enterprise Linux 4, on an x86_64 platform (multi-core Intel Xeon processors, 3.6Ghx, 8GB of RAM). I have some rather complicated code (so I won't attach it here), but it is an iterative algorithm that takes data in the form of an R list, passes the entire list to some compiled C code, converts list items to GSL matrices and vectors, performs its various operations, and sends back the result to R. That result is then sent back to the compiled code until some kind of convergence (I won't bore the group with more details).

The problem is that every .Call to the compiled code runs slower (I print the iteration time using Sys.time() and difftime() ). There is no logical reason for this (i.e., it's not a feature of the algorithm itself). I am using about 20% of my machine's available RAM (it's a large dataset and a memory-intensive algorithm), but there does not appear to be any swapping of memory to disk. I am sure that I am UNPROTECTING the SEXP's that I created, and I am freeing all of the GSL objects at the end of each function. The total RAM used does seem to increase, slowly but steadily, but the speed decrease occurs well before I start coming close to running out of RAM.

Also, it is not just the compiled call that slows down. EVERYTHING slows down, even those that consist only of standard R functions. The time for each of these function calls is roughly proportional to the time of the .Call to the C function.

Another observation is that when I terminate the algorithm, do a rm
(list=ls()), and then a gc(), not all of the memory is returned to the
OS. It is not until I terminate the R session that I get all of the memory back. In my C code, I am not doing anything to de-allocate the SEXP's I create, relying on the PROTECT/UNPROTECT mechanism instead (is this right?).

I spent most of the day thinking I have a memory leak, but that no longer appears to be the case. I tried using Rprof(), but that only gives me the aggregated relative time spent in each function (more than 80% of the time, it's in the .Call).

So I'm stuck. Can anyone help?

Thanks,

Michael

-- 
Michael Braun
Assistant Professor of Marketing
MIT Sloan School of Management
One Amherst St., E40-169
Cambridge, MA 02142

(617) 253-3436
braunm_at_mit.edu ______________________________________________ R-devel_at_r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Fri 25 May 2007 - 23:15:29 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Sat 26 May 2007 - 10:33:10 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.