hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: A proposal for changing pig's memory management
Date Tue, 19 May 2009 22:48:43 GMT
If you have a small number of long-lived large objects and a large number of
small ephemeral objects then the java collector should be in pig-heaven (as
it were).  The long-lived objects will take no time to collect and the
ephemeral objects won't be around to collect by the time the full GC
happens.

On Tue, May 19, 2009 at 3:44 PM, Alan Gates <gates@yahoo-inc.com> wrote:

> Perhaps switching to large buffers instead of having many individual
> objects will address this.  But I'm concerned that if we cannot explicitly
> force data out of memory onto disk then we'll be back in the same boat of
> trusting the Java memory manager.
>

-- 
Ted Dunning, CTO
DeepDyve

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message