[R] Antwort: Re: Antwort: Buying more computer for GLM

From: <g.russell_at_eos-finance.com>
Date: Fri 01 Sep 2006 - 13:33:58 GMT


Prof Brian Ripley wrote:
> I would not have expected glm to be more than say 5x slower than lm if
CPU
> cycles and not memory were the limiting factor. In that case more RAM
> might be all you need.

The ratio between glm and lm might well be about 5x, but that's still a big difference for us. I am pretty sure that RAM is not the main problem; according to the Windows Task Manager the computer is at close to 100% CPU usage, and swapping is not going on. Of course L1/L2 caches may still be
something one can work on, but I'm not sure whether glm has enough repeated access to the same data for that to help. (I don't know how glm works,
but I guess it does a lot of scans through the whole data set, and that the amount of working memory it needs during these scans is basically a function of the number of parameters, not the number of observations, is that right?)

Many thanks for your observations about subset selection by the way, they are a lot of help. Would a good approach be, say, to use some stricter criteria like BIC for choosing a model, and then use non-statistical methods to improve the plausibility of the chosen parameters?

best wishes,

George Russell



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Sat Sep 02 00:52:16 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Thu 07 Sep 2006 - 07:51:17 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.