Re: [Rd] How to execute R scripts simultaneously from multiple threads

From: Erik van Zijst <r_at_erik.prutser.cx>
Date: Thu 04 Jan 2007 - 09:54:35 GMT

Vladimir Dergachev wrote:
> On Wednesday 03 January 2007 3:47 am, Erik van Zijst wrote:
>

>>Appearantly the R C-API does not provide a mechanism for parallel
>>execution..
>>
>>It is preferred that the solution is not based on multi-processing (like
>>C/S), because that would introduce IPC overhead.

>
> One thing to keep in mind is that IPC is very fast in Linux. So unless you are
> making lots of calls to really tiny functions this should not be an issue.

Using pipes or shared memory to pass things around to other processes on the same box is very fast indeed, but if we base our design around something like RServe which uses TCP it could be significantly slower. Our R-based system will be running scripts in response to high-volume real-time stock exchange data, so we expect lots of calls to many tiny functions indeed.

> What can be an issue is the overhead of starting a new R process. In which
> case you can make some helper processes that do the same thing you wanted
> from a multi-thread one and just pass the data around.

Yes, we'll need to keep a pool of processes. Maybe use RServer, or build something ourselves around shared memory if the proof of concept proves TCP is too heavy for this.

thanks,
Erik

> best
>
> Vladimir Dergachev
>
>

>>Hopefully some thread-safe (single-proces) solution is readily
>>available, written in C.
>>
>>What is the best solution to do this?
>>
>>(If there is no single-process solution, what is the alternative?)
>>
>>Regards,
>>Erik.

>
>
-- 
USENET would be a better laboratory is there were more labor and less 
oratory.
		-- Elizabeth Haley

______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Fri Jan 05 03:11:28 2007

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.1.8, at Thu 04 Jan 2007 - 18:31:09 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.