Re: [Rd] R cmd check and multicore foreach loop

From: Renaud Gaujoux <renaud_at_mancala.cbio.uct.ac.za>
Date: Wed, 17 Aug 2011 15:33:59 +0200

Uhm... maybe this is the issue.
The issue seems to specially occur when I am building the vignette, which performs some parallel computations on a reduced example, therefore faster than in a normal usage of the package. The two processes (on my dual core) output some logging information using cat, which are normally sent to the console, but I guess that in the case of a vignette these are written to tex file. It is very few data though (a loop counter), so writing should be also very quick, even in a file.

Could it be possible that a I/O deadlock happens because of this output? I actually use mutexes, at the end of each loop to perform bigger writing operation on a common file, but I hadn't think these would be required for the logging output, assuming that stdout and stderr were thread safe. Maybe I should use mutexes at this level too. Does multicore or doMC provide optional separate logging as doMPI does? (I guess this might be better posted to R-hpc)

Thank you.
Renaud

On 17/08/2011 14:56, Brian G. Peterson wrote:
> On Wed, 2011-08-17 at 04:50 -0700, Tim Triche, Jr. wrote:
>> I'll see if I can put together a self-contained example. Primarily,
>> the times that I use multicore (and attempted to use doSMP, mostly
>> because one of my users refuses to ditch Windows) are when I am
>> reading a ton of binary files, none of which depend on the others.
>> This is a blindingly obvious use-case for e.g. doMC and doSMP, yet
>> what typically happens is that the entire operation wedges. I'm told
>> that doSMP really only works well with Revolution R, but per above, I
>> will try to put together a working self-contained example to show
>> how.
> Remember that physical I/O can bind up the processes too. Having a
> bunch of processes all trying to read from local disk at the same time
> (especially when they are all trying to read the same file(s), a problem
> it seems you may not have) is a recipe for I/O locks that can seize up
> your processes.
>
> So, if the problem only occurs with physical I/O, the first thing I
> would try is to move that storage to a storage device on another machine
> that is tuned for that level of disk I/O.
>
> Regards,
>
> - Brian
>
 

###
UNIVERSITY OF CAPE TOWN This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}



R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel Received on Wed 17 Aug 2011 - 13:50:48 GMT

This quarter's messages: by month, or sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

All messages

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Wed 17 Aug 2011 - 14:00:19 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.

list of date sections of archive