[Rd] as.data.frame requires a lot of memory (PR#14140)

From: <rfalke_at_tzi.de>
Date: Mon, 14 Dec 2009 18:45:15 +0100 (CET)


Full_Name: Raimar Falke
Version: R version 2.10.0 (2009-10-26)
OS: Linux 2.6.27-16-generic #1 SMP Tue Dec 1 19:26:23 UTC 2009 x86_64 GNU/Linux Submission from: (NULL) (134.102.222.56)

The construction of a data frame in the way shown below requires much more memory than expected. If we assume a cell value takes 8 bytes the total amount of the data is 128mb. However the process takes about 920mb and not the expected 256mb (two times the data set).

With the real data sets (~35000 observations with ~33000 attributes) the conversion to a data frame requires has to be killed at with 60gb of memory usage while it should only require 17.6gb (2*8.8gb).

  dfn <- rep(list(rep(0, 4096)), 4096)
  test <- as.data.frame.list(dfn)

I also tried the incremental construction of the data-frame: df$colN <- dataForColN. While I currently can't say much about the memory usage, it takes a looong time.

After the construction the saved-and-loaded data-frame has the expected size.

What is the recommended way to construct larger data-frames?



R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Mon 14 Dec 2009 - 21:21:28 GMT

This archive was generated by hypermail 2.2.0 : Mon 14 Dec 2009 - 22:32:35 GMT