Re: [Rd] data frame subscription operator

From: Vladimir Dergachev <vdergachev_at_rcgardis.com>
Date: Wed 08 Nov 2006 - 16:13:54 GMT

On Wednesday 08 November 2006 3:21 am, Prof Brian Ripley wrote:
>
> > So far I was not able to figure out why this is necessary -
> > could anyone help ?
>
> You need to remove the class to avoid recursion: a few lines later x[i]
> needs to be a call to the primitive and not the data frame method.

I see. Is there a way to get at the primitive directly, i.e. something like `[.list`(x, i) ?

>
> > The reason I am looking at it is that changing attributes forces
> > duplication of the data frame and this is the largest cause of slowness
> > of data.frames in general.
>
> Do you have evidence of that? R has facilities to profile its code, and I
> have never seen [.data.frame taking a significant proportion of the total
> time. If it does for your application, consider if a data frame is an
> appropriate way to store your data. I am not sure we would accept that
> data frames do have 'slowness in general', but their generality does make
> them slower than alternatives where the generality is not needed.

Evidence:

	# this can be copy'n'pasted directly into an R session
	# small N - both system calls return small, but comparable running times
	N<-100000
	A<-data.frame(X=1:N, Y=rnorm(N), Z=as.character(rnorm(N)))
	system.time(B<-A[,1])
	system.time(B<-A[1,1])


	#larger N - both times are larger and still comparable
	N<-1000000
	A<-data.frame(X=1:N, Y=rnorm(N), Z=as.character(rnorm(N)))
	system.time(B<-A[,1])
	system.time(B<-A[1,1])
        

The running times would also grow with the number of columns. Also I have modified 2.4.0 version of R to print out large allocations and I get the impression that the data frame is being duplicated. Same happens for `[<-.data.frame` - but this function has much more complex code, I have not looked through it yet.

Of course, getting a small portion (i.e. A[1:5,]) also takes a lot of time - but the examples showed above should be O(1).

My data is a result of data base query - it has naturally columns of different types and the columns are named (no row.names though) - which is why I used data.frames. What would you suggest ?

                    thank you very much !

                             Vladimir Dergachev

______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Thu Nov 09 03:27:39 2006

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Wed 08 Nov 2006 - 17:30:36 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.