[R] readBin is much slower for raw input than for a file

From: Jon Clayden <jon.clayden_at_gmail.com>
Date: Wed 31 Jan 2007 - 11:03:07 GMT

This hasn't generated any feedback after a few days on R-devel, so I'm forwarding it to R-help in case anyone here has any ideas...


Dear all,

I'm trying to write an efficient binary file reader for a file type that is made up of several fields of variable length, and so requires many small reads. Doing this on the file directly using a sequence of readBin() calls is a bit too slow for my needs, so I tried buffering the file into a raw vector and reading from that ("loc" is the equivalent of the file pointer):

fileSize <- file.info(fileName)$size
connection <- file(fileName, "rb")
bytes <- readBin(connection, "raw", n=fileSize) loc <- 0


# within a custom read function:
if (loc == 0)

    data <- readBin(bytes, what, n, size, ...) else if (loc > 0)

    data <- readBin(bytes[-(1:loc)], what, n, size, ...)

However, this method runs almost 10 times slower for me than the sequence of file reads did. The initial call to readBin() - for reading in the file - is very quick, but running Rprof shows that the vast majority of the run time in doing the full parse is spent in readBin, so it does seem to be that that's slowing things down. Can anyone shed any light on why this is?

I'm not expecting miracles here - and I realise that writing the whole read routine in C would be much quicker - but surely reading from a raw vector should work out faster than reading from a file? The system is R-2.4.1/Linux, Xeon 3.2 GHz, 2 GiB RAM; typical file size is 44 KiB.

Thanks in advance,

R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Wed Jan 31 22:24:14 2007

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Wed 31 Jan 2007 - 13:30:26 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.