Re: [R] can we manage memory usage to increase speed?

From: Zhilin Liu <zhilinliu_email_at_yahoo.com>
Date: Wed 03 Aug 2005 - 07:35:12 EST


Hi,
Thank you all for the kind reply.

I recompiled R as the previous one turned profiling off.

I am using package MAANOVA, running the matest function which is a permutation test. The author did warn that it takes a long time to run.

Here is one of the test results:
[ Rdata]# ./R CMD Rprof maanovatest.out

Each sample represents 0.02 seconds.
Total run time: 905.959999999466 seconds.

Total seconds: time spent in function and callees. Self seconds: time spent in function alone.

   %       total       %       self
 total    seconds     self    seconds    name
100.00    905.94      0.00      0.00     "matest"
 80.18    726.40      0.25      2.30     "fitmaanova"
 79.37    719.04      0.19      1.72     "mixed"
 68.34    619.16      1.05      9.50     "pinv"
 64.33    582.78      0.88      7.96     "La.svd"
 55.51    502.90     55.51    502.90     ".Call"
 38.47    348.54      1.58     14.28     "makeHq"
 34.50    312.60      0.13      1.18     "solveMME"
 19.80    179.42      0.18      1.60    
"matest.engine"
 10.19     92.30     10.19     92.30     "%*%"
......

The other part are not pasted as they are almost the same everytime we check the profiling. Only the parts

above changes. For example, here is another output:
100.00   1411.02      0.00      0.02     "matest"
 82.88   1169.40      0.24      3.40     "fitmaanova"
 82.22   1160.18      0.19      2.74     "mixed"
 68.82    971.02      1.06     14.90     "pinv"
 64.84    914.94      0.85     11.98     "La.svd"
 56.10    791.64     56.10    791.64     ".Call"
 39.13    552.10      1.55     21.84     "makeHq"
 36.77    518.82      0.13      1.88     "solveMME"
 31.40    443.04      0.00      0.00     "matest.perm"
 17.10    241.32      0.16      2.28    
"matest.engine"
 10.15    143.24     10.15    143.24     "%*%"

I run this with a permutation of 2 times and it is still running. So it is not possible to run 1000 permutations with this kind of speed.

And here is the output of TOP for R:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15250 liuz 0 -20 218m 136m 2556 R 72.3 2.5 29:05.20 R

Any suggestion to improve the performance is highly appreciated.

Thanks a lot.

Zhilin

> On 8/2/05, Spencer Graves <spencer.graves@pdf.com>
> wrote:
> > And you can identify inefficient code
> fairly easily taking snapshots
> > from "proc.time" and computing elapsed time for
> sections of your code.
> >
> > spencer graves
>
> Using Rprof may be a better choice. See
>
> ?Rprof
>
> >
> > Tuszynski, Jaroslaw W. wrote:
> >
> > >
> > > If you have a code that takes 2 weeks to run,
> than it might be a case of
> > > inefficient algorithm design. I was able to go
> from overnight runs (SELDI
> > > data analysis) to 20 minute runs by identifying
> single inefficient function
> > > that took most of the time, and writing it in C.
> > >
> > > Jarek
> > >
>

====================================================\=======

> > >
> > > Jarek Tuszynski, PhD.
> o / \
> > > Science Applications International Corporation
> <\__,|
> > > (703) 676-4192
> "> \
> > > Jaroslaw.W.Tuszynski@saic.com
> ` \
> > >
> > >
> > > -----Original Message-----
> > > From: r-help-bounces@stat.math.ethz.ch
> > > [mailto:r-help-bounces@stat.math.ethz.ch] On
> Behalf Of Zhilin Liu
> > > Sent: Monday, August 01, 2005 8:28 PM
> > > To: r-help@stat.math.ethz.ch
> > > Subject: [R] can we manage memory usage to
> increase speed?
> > >
> > > Hi,
> > >
> > > Thanks for reading.
> > >
> > > I am running a process in R for microarray data
> analysis. RedHat Enterprise
> > > Linux 4, dual AMD CPU, 6G memory. However, the R
> process use only a total of
> > > <200M memory. And the CPU usage is total to
> ~110% for two. The program takes
> > > at least 2 weeks to run at the current speed. Is
> there some way we can
> > > increase the usage of CPUs and memories and
> speed up? Any suggestion is
> > > appreciated.
> > >
> > > Thanks again.
> > >
> > > Zhilin
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > ______________________________________________
> > > R-help@stat.math.ethz.ch mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide!
> > > http://www.R-project.org/posting-guide.html
> > >
> > > ______________________________________________
> > > R-help@stat.math.ethz.ch mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> >
> > --
> > Spencer Graves, PhD
> > Senior Development Engineer
> > PDF Solutions, Inc.
> > 333 West San Carlos Street Suite 700
> > San Jose, CA 95110, USA
> >
> > spencer.graves@pdf.com
> > www.pdf.com <http://www.pdf.com>
> > Tel: 408-938-4420
> > Fax: 408-280-7915
> >
> > ______________________________________________
> > R-help@stat.math.ethz.ch mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> >
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>

R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Wed Aug 03 07:43:11 2005

This archive was generated by hypermail 2.1.8 : Sun 23 Oct 2005 - 15:02:09 EST