Re: [R] Applying user function over a large matrix

From: Whit Armstrong <armstrong.whit_at_gmail.com>
Date: Tue, 29 Apr 2008 16:36:41 -0400

are the "chunks" on which you need to apply the function rolling windows? do they overlap?

I have some c++ template utilities that I use for window functions (on timeseries objects) which you are welcome to copy and modify to fit your problem.

they are available here:

git://repo.or.cz/fts.git
git://repo.or.cz/tslib.git

-Whit

On Tue, Apr 29, 2008 at 4:28 PM, Sudipta Sarkar <ssarkar_at_lanworth.com> wrote:

> Hi Jim,
> Thanks for your prompt response,
>
> I am using a fairly powerful Mac with Leopard OS and 17GB RAM
> and 2x3 GhZ intel zeon processor so I do not think the system
> is paging. I also using the Rmpi and snow utilities to
> parallelize it but even then it takes 3.5-4 hours to just
> complete one chunk of matrices.
> You mentioned about storing the data and applying on 1 column
> at a time. Any hint on how I should I go about doing that? I
> cam across the filehash package but am not sure how to use
> apply over an environment variable. So any help in this
> direction will be most welcome.
> thanks
>
> ---- Original message ----
> >Date: Tue, 29 Apr 2008 16:05:41 -0400
> >From: "jim holtman" <jholtman_at_gmail.com>
> >Subject: Re: [R] Applying user function over a large matrix
> >To: "Sudipta Sarkar" <ssarkar_at_lanworth.com>
> >
> >What size machine do you have. A single copy of your object will
> >require 1.5GB of memory. How slow is slow? Is the operating

> system
> >paging because it does not have enough physical memory? can
> you store
> >the data and only operate on 1 column at a time -- this
> reduces the
> >size of the object to 72MB.
> >
> >On Tue, Apr 29, 2008 at 3:16 PM, Sudipta Sarkar
> <ssarkar_at_lanworth.com> wrote:
> >> Respected R experts,
> >> I am trying to apply a user function that basically calls and
> >> applies the R loess function from stat package over each time
> >> series. I have a large matrix of size 21 X 9000000 and I need
> >> to apply the loess for each column and hence I have
> >> implemented this separate user function that applies loess
> >> over each column and I am calling this function foo as follows:
> >> xc<-apply(t,2,foo) where t is my 21 X 9000000 matrix and
> >> loess. This is turning out to be a very slow process and I
> >> need to repeat this step for 25-30 such large matrix chunks.
> >> Is there any trick I can use to make this work faster?
> >> Any help will be deeply appreciated.
> >> Regards
> >>
> >>
> >> Sudipta Sarkar PhD
> >> Senior Analyst/Scientist
> >> Lanworth Inc. (Formerly Forest One Inc.)
> >> 300 Park Blvd., Ste 425
> >> Itasca, IL
> >> Ph: 630-250-0468
> >>
> >> ______________________________________________
> >> R-help_at_r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained,
> reproducible code.
> >>
> >
> >
> >
> >--
> >Jim Holtman
> >Cincinnati, OH
> >+1 513 646 9390
> >
> >What is the problem you are trying to solve?
>
>
> Sudipta Sarkar PhD
> Senior Analyst/Scientist
> Lanworth Inc. (Formerly Forest One Inc.)
> 300 Park Blvd., Ste 425
> Itasca, IL
> Ph: 630-250-0468
>
> ______________________________________________
> R-help_at_r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]



R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Tue 29 Apr 2008 - 21:03:32 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Tue 29 Apr 2008 - 21:30:39 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive