Re: [Rd] idea for "virtual matrix/array" class

From: Prof Brian Ripley <>
Date: Tue 24 Aug 2004 - 07:23:02 EST

We've seen something very similar before. A file is just a database writ small, and RPgSQL did essentially this in ca 2001. Fei Chen's research system (see his talk at DSC2003) again does similar things.

Both of those were for virtual data frames as I recall. It's easier for a matrix/array, but I am not sure it is necessary. At least on Unix-alikes one could arrange to memory-map large vectors (logical, integer, double or raw). Character vectors would be a little harder, but a char ** version could be memory mapped. I presume you don't mean matrix/arrays of lists?

I think you need to show examples where an array/matrix is the right structure and the sort of access required. Accessing rows in a matrix would cause severe page-faulting for example (and that's the sort of reason we recommend DBMSes).

On Mon, 23 Aug 2004, Tony Plate wrote:

> I've been wondering how to work with more data than can fit in memory, in a
> way that allows it to be worked with conveniently and quickly. Of course,
> a database can be used for this purpose, but extracting data from a
> database is much slower and somewhat less convenient than extracting data
> from a native object (at least in our setup).
> One idea I was thinking about was to have a new class of object that
> referred to data in a file on disk, and which had all the standard methods
> of matrices and arrays, i.e., subsetting ("["), dim, dimnames, etc. The
> object in memory would only store the array attributes, while the actual
> array data (the elements) would reside in a file. When some extraction
> method was called, it would access data in the file and return the
> appropriate data. With sensible use of seek operations, the data access
> could probably be quite fast. The file format of the object on disk could
> possibly be the standard serialized binary format as used in .RData
> files. Of course, if the object was larger than would fit in memory, then
> trying to extract too large a subarray would exhaust memory, but it should
> be possible to efficiently extract reasonably sized subarrays. To be more
> useful, one would want want apply() to work with such arrays. That would
> be doable, either by creating a new method for apply, or possibly just for
> aperm.
> Some difficulties that might arise could have to do with functions like
> "typeof" and "mode" and "storage.mode" -- what should these return for such
> an object? Such a class would probably break the common relationships such
> as x[1] having the same storage mode as x. I don't know if difficulties
> like these would ultimately make such a "virtual array" class unworkable.
> Does anyone have any opinions as to the merits of this idea? Would there
> be any interest in seeing such a class in R?

Brian D. Ripley,        
Professor of Applied Statistics,
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________ mailing list
Received on Tue Aug 24 07:34:03 2004

This archive was generated by hypermail 2.1.8 : Wed 03 Nov 2004 - 22:45:08 EST