Re: [Rd] gc()$Vcells < 0 (PR#9345)

From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk>
Date: Tue 07 Nov 2006 - 11:28:55 GMT

On Mon, 6 Nov 2006, Vladimir Dergachev wrote:

> On Monday 06 November 2006 6:12 pm, dmaszle@mendelbio.com wrote:
>> version.string Version 2.3.0 (2006-04-24)
>>
>>> x<-matrix(nrow=44000,ncol=48000)
>>> y<-matrix(nrow=44000,ncol=48000)
>>> z<-matrix(nrow=44000,ncol=48000)
>>> gc()
>>
>> used (Mb) gc trigger (Mb) max used (Mb)
>> Ncells 177801 9.5 407500 21.8 350000 18.7
>> Vcells -1126881981 24170.6 NA 24173.4 NA 24170.6
>>
>
> Happens to me with versions 2.40 and 2.3.1. The culprit is this line
> in src/main/memory.c:
>
> INTEGER(value)[1] = R_VSize - VHEAP_FREE();
>
> Since the amount used is greater than 4G and INTEGER is 32bit long
> (even on 64 bit machines) this returns (harmless) nonsense.

That's not quite correct. The units here are Vcells (8 bytes), and integer() is signed, so this can happen only if more than 16Gb of heap is allocated.

We are aware that we begin to hit problems at 16Gb: it is for example the maximum size of an R vector. Those objects are logical and so about 7.8Gb each: their length as vectors is 98% of the maximum possible. However, the first time we discussed it we thought it would be about 5 years before those limits would become important -- I think three of those years have since passed.

> The megabyte value nearby is correct and gc trigger and max used fields are
> marked as NA already.

and now 'used' is also marked as NA in 2.4.0 patched.

This is only a reporting issue. When I first used R it reported only numbers, and I added the Mb as a more comprehensible figure (especially for Ncells). I think it would be sensible now to only report these figures in Mb or Gb (and also the reports for gcinfo(TRUE)).

The model behind the report actually pre-dates the GC change in 1.2.0. The 'Vcells' are nowadays the sum of all the allocations from VECSXPs (which include their headers), rather than the 'vector heap' (although some of the earlier terminology persists).

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Tue Nov 07 22:31:45 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Tue 07 Nov 2006 - 16:30:35 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.