Re: [Rd] Arrays Partial unserialization

From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk>
Date: Fri, 31 Aug 2012 15:51:51 +0100

On 31/08/2012 15:41, Duncan Murdoch wrote:
> On 31/08/2012 9:47 AM, Damien Georges wrote:
>> Hi all,
>>
>> I'm working with some huge array in R and I need to load several ones to
>> apply some functions that requires to have all my arrays values for each
>> cell...
>>
>> To make it possible, I would like to load only a part (for example 100
>> cells) of all my arrays, apply my function, delete all cells loaded,
>> loaded following cells and so on.
>>
>> Is it possible to unserialize (or load) only a defined part of an R
>> array ?
>> Do you know some tools that might help me?
>
> I don't know of any tools to do that, but there are tools to maintain
> large objects in files, and load only parts of them at a time, e.g. the
> ff package. Or you could simply use readBin and writeBin to do the same
> yourself.

Serialization is essentially serial, so you can only read the serialized format from the beginning. So too are the compression algorithms used by default.

>> Finally, I did lot of research to find the way array (and all other R
>> object) are serialized into binary object, but I found nothing
>> explaining really algorithms involved. If someone has some information
>> on this topic, I'm interesting in.
>
> You can read the source for this; it is in src/main/serialize.c.

And there is an extensive commentary in the 'R Internals' manual.

-- 
Brian D. Ripley,                  ripley_at_stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Fri 31 Aug 2012 - 14:54:52 GMT

This quarter's messages: by month, or sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

All messages

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Sat 01 Sep 2012 - 00:50:41 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.

list of date sections of archive