Re: [R] RMySQL query: why result takes so much memory in R ?

From: Duncan Murdoch <murdoch_at_stats.uwo.ca>
Date: Tue 03 May 2005 - 01:18:35 EST

Christoph Lehmann wrote:
> Hi
> I just started with RMySQL. I have a database with roughly 12 millions
> rows/records and 8 columns/fields.
>
> From all 12 millions of records I want to import 3 fields only.
> The fields are specified as:id int(11), group char(15), measurement
> float(4,2).
> Why does this take > 1G RAM? I run R on suse linux, with 1G RAM and with
> the code below it even fills the whole 1G of swap. I just don't
> understand how 12e6 * 3 can fill such a huge range of RAM? Thanks for
> clarification and potential solutions.

Those fields are each 8 or 20 bytes in size, so you're talking 12e6 times 36 or about nearly half a Gig for each copy. Presumably the code is storing more than one or two copies of the data.

Why don't you use fetch() to get your records in more manageable chunks?

Duncan Murdoch

>
>
> ## my code
> library(RMySQL)
> drv <- dbDriver("MySQL")
> ch <- dbConnect(drv,dbname="testdb",
> user="root",password="mysql")
> testdb <- dbGetQuery(ch,
> "select id, group, measurement from mydata")
> dbDisconnect(ch)
> dbUnloadDriver(drv)
>
> ## end of my code
>
> Cheers
> Christoph
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html



R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Received on Tue May 03 01:46:14 2005

This archive was generated by hypermail 2.1.8 : Fri 03 Mar 2006 - 03:31:31 EST