RE: [R] naive question

From: Vadim Ogranovich <vograno_at_evafunds.com>
Date: Fri 02 Jul 2004 - 11:49:49 EST


Richard,

 Thank you for the analysis. I don't think there is an inconsistency between the factor of 4 you've found in your example and 20 - 50 I found in my data. I guess the major cause of the difference lies with the structure of your data set. Specifically, your test data set differs from mine in two respects:
* you have fewer lines, but each line contains many more fields (12500 * 800 in your case and 3.8M * 10 in my)
* all of your data fields are doubles, not strings. I have a mixture of doubles and strings.

I posted a more technical message to r-devel where I discussed possible reasons for the IO slowness. One of them is that R is slow at making strings. So if you try to read your data as strings, colClasses=rep("character", 800), I'd guess you will see a very different timing. Even simple reshaping of your matrix, say make it (12500*80) rows by 10 columns, will considerably worsen it. Please let me know the results if you do anything of the above.

In my message to r-devel you may also find some timing that supports my estimates.

Thanks,
Vadim

> -----Original Message-----
> From: r-help-bounces@stat.math.ethz.ch
> [mailto:r-help-bounces@stat.math.ethz.ch] On Behalf Of
> Richard A. O'Keefe
> Sent: Thursday, July 01, 2004 5:22 PM
> To: r-help@stat.math.ethz.ch
> Subject: RE: [R] naive question
>
> As part of a continuing thread on the cost of loading large
> amounts of data into R,
>
> "Vadim Ogranovich" <vograno@evafunds.com> wrote:
> R's IO is indeed 20 - 50 times slower than that of
> equivalent C code
> no matter what you do, which has been a pain for some of us.
>
> I wondered to myself just how bad R is at reading, when it is
> given a fair chance. So I performed an experiment.
> My machine (according to "Workstation Info") is a SunBlade
> 100 with 640MB of physical memory running SunOS 5.9 Generic,
> according to fpversion this is an Ultra2e with the CPU clock
> running at 500MHz and the main memory clock running at 84MHz
> (wow, slow memory). R.version is platform sparc-sun-solaris2.9
> arch sparc
> os solaris2.9
> system sparc, solaris2.9
> status
> major 1
> minor 9.0
> year 2004
> month 04
> day 12
> language R
> and althnough this is a 64-bit machine, it's a 32-bit
> installation of R.
>
> The experiment was this:
> (1) I wrote a C program that generated 12500 rows of 800 columns, the
> numbers were integers 0..999,999,999 generated using drand48().
> These numbers were written using printf(). It is possible to do
> quite a bit better by avoiding printf(), but that would ruin the
> spirit of the comparison, which is to see what can be done with
> *straightforward* code using *existing* library functions.
>
> 21.7 user + 0.9 system = 22.6 cpu seconds; 109 real seconds.
>
> The sizes were chosen to get 100MB; the actual size was
> 12500 (lines) 10000000 (words) 100012500 (bytes)
>
> (2) I wrote a C program that read these numbers using
> scanf("%d"); it
> "knew" there were 800 numbers per row and 12500 numbers in all.
> Again, it is possible to do better by avoiding scanf(), but the
> point is to look at *straightforward* code.
>
> 18.4 user + 0.6 system = 19.0 cpu seconds; 100 real seconds.
>
> (3) I started R, played around a bit doing other things, then
> issued this
> command:
>
> > system.time(xx <- read.table("/tmp/big.dat",
> header=FALSE, quote="",
> + row.names=NULL, colClasses=rep("numeric",800), nrows=12500,
> + comment.char="")
>
> So how long _did_ it take to read 100MB on this machine?
>
> 71.4 user + 2.2 system = 73.5 cpu seconds; 353 real seconds.
>
> The result: the R/C ratio was less than 4, whether you
> measure cpu time or real time. It certainly wasn't anywhere
> near 20-50 times slower.
>
> Of course, *binary* I/O in C *would* be quite a bit faster:
> (1') generate same integers but write a row at a time using fwrite():
> 5 seconds cpu, 25 seconds real; 40 MB.
>
> (2') read same integers a row at a time using fread()
> 0.26 seconds cpu, 1 second real.
>
> This would appear to more than justify "20-50 times slower",
> but reading binary data and reading data in a textual
> representation are different things, "less than 4 times
> slower" is the fairer measure. However, it does emphasise
> the usefulness of problem-specific bulk reading techniques.
>
> I thought I'd give you another R measurement:
> > system.time(xx <- read.table("/tmp/big.dat", header=FALSE))
> But I got sick of waiting for it, and killed it after 843 cpu seconds,
> 3075 real seconds. Without knowing how far it had got, one
> can say no more than that this is at least 10 times slower
> than the more informed call to read.table.
>
> What this tells me is that if you know something about the
> data that you _could_ tell read.table about, you do yourself
> no favour by keeping read.table in the dark. All those
> options are there for a reason, and it *will* pay to use them.
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
>



R-help@stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Fri Jul 02 11:53:33 2004

This archive was generated by hypermail 2.1.8 : Wed 03 Nov 2004 - 22:54:39 EST