Re: [Rd] Suggestion for serialization performance improvement on Windows

From: Prof Brian Ripley <>
Date: Tue, 20 Jul 2010 11:45:31 +0100 (BST)

On Fri, 9 Jul 2010, Bryan W. Lewis wrote:

> Dear R developers,
> The slow performance of serializing to a raw vector on Windows is an
> issue that has appeared in this list before. It appears to be due to


> the frequent use of realloc from the resize_buffer method in
> serialize.c.
> I suggest a more granular, but still incremental, re-allocation of
> memory. For example change near the top of resize_buffer to:
> R_size_t newsize = needed + 65536 - (needed % 65536);
> or some other similar small multiple of a typical system page size.

for some definition of 'small multiple'

> I have found this to dramatically improve performance of serialization
> to raw vectors on Windows.

However, I didn't and you presented no evidence. On HB's 2008 example your idea achieved for me a speedup of about 3x. A much better speedup (15x) was achieved by switching serialize.c to use the alternative malloc used by memory.c, and using a much larger page size (e.g. 1Mb) was better still. But changing the re-allocation strategy resulted in a 150x speed up, to levels comparable to decent operating systems like Linux and Solaris with the existing code.

(In case it matters, I was using x64 Windows 7.)

Ideally you would have

So please test R-devel, and if there is still a problem reply with all the details omitted here.

Brian D. Ripley,        
Professor of Applied Statistics,
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________ mailing list
Received on Tue 20 Jul 2010 - 10:51:30 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Tue 20 Jul 2010 - 13:20:17 GMT.

Mailing list information is available at Please read the posting guide before posting to the list.

list of date sections of archive