Re: [Rd] RSQLite indexing

From: hadley wickham <h.wickham_at_gmail.com>
Date: Mon, 22 Oct 2007 17:37:57 -0500

On 10/22/07, Thomas Lumley <tlumley_at_u.washington.edu> wrote:
>
> I am trying to use RSQLite for storing data and I need to create indexes on
> two variables in the table. It appears from searching the web that the CREATE
> INDEX operation in SQLite is relatively slow for large files, and this has been
> my experience as well.
>
> The two index variables are crossed. One has about 350,000 levels [yes, it's
> genetic association data]. The other will have about 4000 levels eventually,
> but is up to about 100 now. When the data were entered they were already ordered by this second index variable.
>
> Creating the index took about an hour on the 100-level, presorted variable and about 12 hours on the 350,000-level unsorted variable. I'm looking for advice on how to reduce this. Specifically

How big is your dataset? SQLite can be slow, but I didn't think it was that slow.

> 1/ would it be faster if the variable with more levels was the presorted one?

I didn't think this would matter (and can imagine situations where it would be worse). Is there a particular reason you think this might help?

> 2/ would it be faster or slower if the index were created before adding all the data?

It's generally much faster to create the index after loading all the data.

> 3/ are there any options that can be set to speed up the indexing?

Have you tried 'vacuum'ing your database prior to indexing?

Will you be indexing on both columns simultaneously? If so, you might try creating a single index.

Hadley

-- 
http://had.co.nz/

______________________________________________
R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Mon 22 Oct 2007 - 22:41:51 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Thu 25 Oct 2007 - 11:37:11 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.