Re: [R] advice requested re: building "good" system (R, SQL db) for handling large datasets

From: Richard Pearson <richard.pearson_at_postgrad.manchester.ac.uk>
Date: Wed, 06 Feb 2008 12:25:43 +0000

Hi Thomas

I'm certainly no expert but thought I'd reply as I'm likely to be in a similar position soon.

With regards versions of R I think you should always have the latest release version. This will mean upgrading at least every 6 months, but this shouldn't be too much of a problem.

With OSs, you need to be aware that there is an upper limit to the amount of RAM than be handled (2-4 GB) with many. I think if you plan to use more than 4GB RAM, you should definitely consider 64-bit linux. I have no information or opinions as to which flavour of linux.

With databases, one issue that might be relevant is whether you want to store data in tables (e.g. one table to store one data.frame) that can subsequently be manipulated in the DB, or to store R objects as R objects (e.g. as BLOBs). My situation is likely to be the later case, and one of my concerns is that many DBs have an upper limit of 2GB on BLOBs, and I might potentially have objects that are larger than this.

Finally, you might get more response on database issues from R-sig-db than R-help.

Best wishes

Richard.

Thomas Pujol wrote:
> R-community,
> Sometime during the next 12-months, I plan on configuring a new computer system on which I will primarily run "R" and a SQL database (Microsoft SQL Server, MySQL, Oracle, etc). My primary goal is to "optimize" the system for R, and for passing data to and from R and the database.
>
> I work with large datasets, and therefore I "think" one of my most important goals should be to maximize the amount of RAM that R can utilize effectively.
>
> I am seeking advice concerning the version of R, OS, processor, hard-drive/storage configuration, database, etc. that I should consider. (I am guessing that I should build a system with lots of RAM, and a Linux OS, but am seeking advice from the R community.) If I choose Linux, does it matter which version I use? Any opinion regarding implementing a commercially supported version from a vendor such as Red Hat, Sun, etc? Is any database particularly better at "exchanging" data with R?
>
> While cost is of course a consideration, it is probably a secondary consideration to overall performance, reliability, and ease of ongoing maintenance/support.
>
> Thanks!
>
>
> ---------------------------------
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help_at_r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>



R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Wed 06 Feb 2008 - 12:28:27 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Wed 06 Feb 2008 - 16:30:12 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive