Re: [R] help: cannot allocate vector of length 828310236

From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk>
Date: Tue 15 Aug 2006 - 16:27:36 EST

Does it make any statistical sense to do polr or probit regression (not the same thing) on `really huge data'? There are few regression-like problems in which model inadequacy does not swamp estimation uncertainty for as few as a 1000 cases.

If you want to do that sort of thing, by all means use SAS to do it. But if you are not prepared to spend a few $$ on adequate RAM, don't expect free technical consultancy, especially not from those whose work you are using and not crediting.

On Mon, 14 Aug 2006, T Mu wrote:

> Hi all,
>
> I was trying a probit regression using polr() and got this message,

polr is a strange choice of tool for 'probit regression' as the term is usually used. It does 'ordered probit regression'.

> Error in model.matrix.default(Terms, m, contrasts) :
> cannot allocate vector of length 828310236
>
> The data is about 20M (a few days ago I asked a question about large file,
> thank you for responses, then I use MS Access to select those columns I
> would use).
>
> R is 2.3.1, Windows XP, 512M Ram.
>
> I am going to read some help on memory use in R, but hope anybody can give
> me some quick hints.

Quick hint: read and follow the posting guide BEFORE posting.

> Is it because iphysical memory runs out, or some other things could be wrong
> with data or polr()?
> Does R use virtual memory? If so, what options can I set?
> If not, can R deal with really huge data (except adding RAM according to
> data size)? If this is the case, it is too bad that I have to tell my boss
> to go back to SAS. Now it is not a speed issue yet.
>
> Thank you.
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Received on Tue Aug 15 16:37:52 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Tue 15 Aug 2006 - 18:22:36 EST.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.