Re: [Rd] 4-int indexing limit of R {Re: [R] allocMatrix limits}

From: Vadim Kutsyy <vadim_at_kutsyy.com>
Date: Fri, 01 Aug 2008 10:22:43 -0700

Martin Maechler wrote:
> [[Topic diverted from R-help]]
>

> Well, fortunately, reasonable compilers have indeed kept
> 'long' == 'long int' to mean 32-bit integers
> ((less reasonable compiler writers have not, AFAIK: which leads
> of course to code that no longer compiles correctly when
> originally it did))
> But of course you are right that 64-bit integers
> (typically == 'long long', and really == 'int64') are very
> natural on 64-bit architectures.
> But see below.
>
well in 64bit Ubunty, /usr/include/limits.h defines:

/* Minimum and maximum values a `signed long int' can hold. */

#  if __WORDSIZE == 64
#   define LONG_MAX     9223372036854775807L
#  else
#   define LONG_MAX     2147483647L
#  endif
#  define LONG_MIN      (-LONG_MAX - 1L)

and using simple code to test
(http://home.att.net/~jackklein/c/inttypes.html#int) my desktop, which is standard Intel computer, does show.

Signed long min: -9223372036854775808 max: 9223372036854775807

> If you have too large a numeric matrix, it would be larger than
> 2^31 * 8 bytes ~= 2^34 / 2^20 ~= 16'000 Megabytes.
> If that is is 10% only for you, you'd have around 160 GB of
> RAM. That's quite a impressive.

>
 > cat /proc/meminfo | grep MemTotal
MemTotal: 145169248 kB

We have "smaller" SGI NUMAflex to play with, where the memory can increased to 512Gb ("larger" version doesn't have this "limitation"). But with even commodity hardware you can easily get 128Gb for reasonable price (i.e. Dell PowerEdge R900)

> Note that R objects are (pointers to) C structs that are
> "well-defined" platform independently, and I'd say that this
> should remain so.
>
>
I forgot that R stores two dimensional array in a single dimensional C array. Now I understand why there is a limitation on total number of elements. But this is a big limitations.
> One of the last times this topic came up (within R-core),
> we found that for all the matrix/vector operations,
> we really would need versions of BLAS / LAPACK that would also
> work with these "big" matrices, ie. such a BLAS/Lapack would
> also have to internally use "longer int" for indexing.
> At that point in time, we had decied we would at least wait to
> hear about the development of such BLAS/LAPACK libraries
BLAS supports two dimensional metrics definition, so if we would store matrix as two dimensional object, we would be fine. But than all R code as well as all packages would have to be modified.



R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Fri 01 Aug 2008 - 17:32:15 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Fri 01 Aug 2008 - 22:35:57 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.

list of date sections of archive