[Rd] readBin is much slower for raw input than for a file

From: Jon Clayden <jon.clayden_at_gmail.com>
Date: Fri 26 Jan 2007 - 11:25:00 GMT

Dear all,

I'm trying to write an efficient binary file reader for a file type that is made up of several fields of variable length, and so requires many small reads. Doing this on the file directly using a sequence of readBin() calls is a bit too slow for my needs, so I tried buffering the file into a raw vector and reading from that ("loc" is the equivalent of the file pointer):

fileSize <- file.info(fileName)$size
connection <- file(fileName, "rb")
bytes <- readBin(connection, "raw", n=fileSize) loc <- 0


# within a custom read function:
if (loc == 0)
    data <- readBin(bytes, what, n, size, ...)
else if (loc > 0)
    data <- readBin(bytes[-(1:loc)], what, n, size, ...)

However, this method runs almost 10 times slower for me than the
sequence of file reads did. The initial call to readBin() - for
reading in the file - is very quick, but running Rprof shows that the
vast majority of the run time in doing the full parse is spent in
readBin, so it does seem to be that that's slowing things down. Can
anyone shed any light on why this is?

I'm not expecting miracles here - and I realise that writing the whole
read routine in C would be much quicker - but surely reading from a
raw vector should work out faster than reading from a file? The system
is R-2.4.1/Linux, Xeon 3.2 GHz, 2 GiB RAM; typical file size is 44

Thanks in advance,

R-devel@r-project.org mailing list
Received on Fri Jan 26 22:28:29 2007

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Fri 26 Jan 2007 - 13:31:15 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.