Re: [R] parallel processing an lme model

From: Douglas Bates <bates_at_stat.wisc.edu>
Date: Tue, 29 May 2007 12:44:48 -0500

On 5/28/07, toby909_at_gmail.com <toby909_at_gmail.com> wrote:
> Hi All

> Had anyone of you seen if it is possible to split a large lme() job to be
> processed by multiple cpus/computers?

> I am just at the very beginning to understand related things, but does the lme()
> use solution finding functions like nlm() mle(), and I-dont-know-what-else of
> the standard R pagage or does lme come with its own? If the former, has onyone
> seen how to split an mle() function call to be processed by multiple cpus?

First, if you want speed and your model can be fit by lmer I would recommend using lmer or lmer2 from the lme4 package. These functions can fit models with crossed or partially crossed random effects which is often the case for models in very large data sets. However, they do not provide the facility for specifying correlation structures or variance functions in addition to those implied by the random effects.

Both lme and lmer end up calling nlminb to do the optimization of the log-likelihood or the REML criterion. The lmer2 function does not call nlminb explicitly but does use the underlying code from nlminb.

None of these operations are easily parallelizable. The only hope for getting a speed boost from multiple CPU cores or multiple processors is by using a multithreaded accelerated BLAS (Basic linear algebra subroutines) library (see the R Installation and Administration manual for details). However, in some cases we have observed that multithreaded BLAS actually slow down the computation. To check if this is the case for you try the following both with and without multithreaded BLAS.

library(lme4)
data(star, package = "mlmRev")
system.time(fm1 <- lmer(math ~ sx*eth+ses+gr+cltype

   +(yrs|id)+(1|tch)+(yrs|sch),
    star, control = list(grad = 0, nit = 0, msV = 1)) system.time(m1 <- lmer2(math ~ sx*eth+ses+gr+cltype

   +(yrs|id)+(1|tch)+(yrs|sch),
    star, control = list(msV = 1))

> In the case if, I would very much appreicate your hint or point to a source
> where I can read about.



R-help_at_stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Tue 29 May 2007 - 18:13:44 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Tue 29 May 2007 - 18:31:17 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.