hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pi song" <pi.so...@gmail.com>
Subject Re: Spillable memory manager
Date Fri, 23 May 2008 14:07:16 GMT
Dear Tanton,

This means the MemoryManager is not successful at reclaiming memory.  Did
that happen in Map phase or Reduce phase? Did you use a custom chunker? How
big is your split?

Pi

On Fri, May 23, 2008 at 3:52 PM, Tanton Gibbs <tanton.gibbs@gmail.com>
wrote:

> I upgraded to hadoop 17 and the latest Pig from svn.
>
> I'm now getting a ton of lines in my log files that say:
>
> 2008-05-23 00:49:27,832 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 1441792(1408K) used = 483176072(471851K) committed =
> 641335296(626304K) max = 954466304(932096K)
>
> In addition, jobs on big files are running very slowly.
>
> Does anyone have any ideas as to what I could have screwed up?
>
> Thanks!
> Tanton
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message