[Rd] looking for adice on bigmemory framework with C++ and java interoperability

From: andre zege <andre.zege_at_gmail.com>
Date: Fri, 04 May 2012 21:50:22 -0400

I work with problems that have rather large data requirements -- typically a bunch of multigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked matrices from C++, and ideally java without being called from R. Having ability to save armadillo matrices into filebacked matrices and load them back into armadillo would be another very useful thing. This would allow really smooth cooperation between various pieces of software. I would prefer to avoid using Rinside for that.

I guess i could hack bigmemory C++ code a bit, compile it into a C++ shared library and it'll do. I guess i could hack it a bit to work with armadillo matrices as well. I don't want however to reinvent the wheel and if there is something like that already somewhere i would rather use it for the moment. Looking very much for suggestions. If there is truly nothing like that and someone with C++ or especially java development experience is interested and want to cooperate on this, let me know too.


NB. I guess something like what i want -- access to the same disc caches from R, C++, and java (and python) exists in HDF world. I, however, don't know how performance of HDF compares with bigmemory matrices, which i come to like and appreciate a lot. If there is someone who could address simplicity of use and performance of HDF vs bigmemory, it'd be very interesting.

        [[alternative HTML version deleted]]

R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Sat 05 May 2012 - 01:52:17 GMT

This quarter's messages: by month, or sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

All messages

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Sat 05 May 2012 - 23:10:56 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.

list of date sections of archive