Re: [Rd] There was a problem by the use of snow.

From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk>
Date: Mon, 25 Jun 2007 11:18:11 +0100 (BST)

On Mon, 25 Jun 2007, Ei-ji Nakama wrote:

> problem of the very large memory require by the Sign extension.
>
> --- R-2.5.0.orig/src/main/serialize.c 2007-03-27 01:42:08.000000000 +0900
> +++ R-2.5.0/src/main/serialize.c 2007-06-25 00:48:58.000000000 +0900
> @@ -1866,7 +1866,7 @@
>
> static void resize_buffer(membuf_t mb, int needed)
> {
> - int newsize = 2 * needed;
> + size_t newsize = 2 * needed;
> mb->buf = realloc(mb->buf, newsize);
> if (mb->buf == NULL)
> error(_("cannot allocate buffer"));

Yes, thanks, but the structure also needs to be changed as the next line is

     mb->size = newsize;

and so this would set mb->size to a negative value.

Could you please tell us where you encountered this?

As far as I can see, this code is only used via R_serialize for serializing to a raw vector, in which case 'size' can safely be 'int' and the re-allocation should be up to 2^31-1 bytes at most (and allocating twice what you are asked for seems undesirable). But there is potential overflow at

     if (mb->count + length > mb->size)

and possibly elsewhere.

> The time-out of read and write was not set.
>
> 51:sendData.SOCKnode <- function(node, data) {
> 52: timeout <- getClusterOption("timeout")
> 53: old <- options(timeout = timeout);
> 54: on.exit(options(old))
> 55: serialize(data, node$con)
> 56: }
> 57:
> 58:recvData.SOCKnode <- function(node) {
> 59: timeout <- getClusterOption("timeout")
> 60: old <- options(timeout = timeout);
> 61: on.exit(options(old))
> 62: unserialize(node$con)
> 63: }

I don't think sock_read/sock_write is the right place to make that setting. Ideally it would be set when the option is set, but as this is in a module that needs an extension to the interface.

Looking at the code, we read from a socket in blocks of 4096, but we write in a single block. The latter is likely to be the problem, and I think some redesigning is necessary here.

Perhaps you and Luke Tierney can comment on exactly what the problem is and how best to work around it.

>
> --- R-2.5.0.orig/src/modules/internet/sockconn.c 2006-09-04 23:20:59.000000000 +0900
> +++ R-2.5.0/src/modules/internet/sockconn.c 2007-06-25
> 00:51:38.000000000 +0900
> @@ -155,14 +155,19 @@
> static size_t sock_read(void *ptr, size_t size, size_t nitems,
> Rconnection con)
> {
> + int timeout = asInteger(GetOption(install("timeout"), R_BaseEnv));
> +
> + R_SockTimeout(timeout);
> return sock_read_helper(con, ptr, size * nitems)/size;
> }
>
> static size_t sock_write(const void *ptr, size_t size, size_t nitems,
> Rconnection con)
> {
> + int timeout = asInteger(GetOption(install("timeout"), R_BaseEnv));
> Rsockconn this = (Rsockconn)con->private;
>
> + R_SockTimeout(timeout);
> return R_SockWrite(this->fd, ptr, size * nitems)/size;
> }
>
>

-- 
Brian D. Ripley,                  ripley_at_stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-devel_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Received on Mon 25 Jun 2007 - 10:20:55 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Mon 25 Jun 2007 - 13:35:31 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-devel. Please read the posting guide before posting to the list.