Re: [R] Need help: Loading a large data set.

From: jim holtman <jholtman_at_gmail.com>
Date: Sun, 23 Nov 2008 16:04:26 -0500

What matrix package are you using? I have not used sparse matrices, but a quick look at the help file of Matrix talks about a file format for reading in a sparse matrix. I would assume that all you need to do is to read in your files and write them out in that format. You can do it in using 'list.files' to read in the files and the 'cat' (or any other command that will write an ASCII file) to output the data in the correct format.

On Sun, Nov 23, 2008 at 3:19 PM, Atul Kulkarni <atulskulkarni_at_gmail.com> wrote:
> Hi All,
>
> I am dealing with a large data set which translates in to a sparse matrix, I
> want to load this data that is spread in approximately 17000+ files each
> defining a row and each file has variable number of records that come with
> its column number and the value that they store.
>
> I wish to load this data in memory from these files one by one. Is there
> anyway I can do this in R, before I start processing? I am sure this is not
> the first time R or the community is confronted with this kind of a problem
> but I could not find the documentation for loading data in to sparse matrix
> I found quite a few packages for sparse matrix but they all were
> concentrating on how to do various operations with the matrix once the
> matrix is loaded. I need to first load the data in the system before I can
> think about analysing.
>
> Regards,
> Atul.
>
> Graduate Student,
> Department of Computer Science,
> University of Minnesota Duluth,
> Duluth, MN, 55812.
> --------
> www.d.umn.edu/~kulka053
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help_at_r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?

______________________________________________
R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Received on Sun 23 Nov 2008 - 21:08:58 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Sun 23 Nov 2008 - 21:30:26 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive