hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: A proposal for changing pig's memory management
Date Fri, 15 May 2009 02:43:53 GMT
That Telegraph dataflow paper is pretty long in the tooth.  Certainly
several of their claims have little force any more (lack of non-blocking
I/O, poor thread performance, no unmap, very expensive synchronization for
uncontested locks).  It is worth that they did all of their tests on the 1.3
JVM and things have come an enormous way since then.

Certainly, it is worth having opaque contains based on byte arrays, but
isn't that pretty much what the NIO byte buffers are there to provide?
Wouldn't a virtual tuple type that was nothing more than a byte buffer, type
and an offset do almost all of what is proposed here?

On Thu, May 14, 2009 at 5:33 PM, Alan Gates <gates@yahoo-inc.com> wrote:

> http://wiki.apache.org/pig/PigMemory
>
> Alan.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message