giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Peters <jeffpeters1...@gmail.com>
Subject Re: Out of memory with giraph-release-1.0.0-RC3, used to work on old Giraph
Date Thu, 29 Aug 2013 00:33:18 GMT
Ok thanks Avery, I'll try it. I'm not sure I know how I would do that on a
running AWS EMR instance, but I can do it on a local stand-alone Hadoop
running a smaller version of the job and see if anything jumps out.


On Wed, Aug 28, 2013 at 4:57 PM, Avery Ching <aching@apache.org> wrote:

> Try dumping a histogram of memory usage from a running JVM and see where
> the memory is going.  I can't think of anything in particular that
> changed...
>
>
> On 8/28/13 4:39 PM, Jeff Peters wrote:
>
>>
>> I am tasked with updating our ancient (circa 7/10/2012) Giraph to
>> giraph-release-1.0.0-RC3. Most jobs run fine but our largest job now runs
>> out of memory using the same AWS elastic-mapreduce configuration we have
>> always used. I have never tried to configure either Giraph or the AWS
>> Hadoop. We build for Hadoop 1.0.2 because that's closest to the 1.0.3 AWS
>> provides us. The 8 X m2.4xlarge cluster we use seems to provide 8*14=112
>> map tasks fitted out with 2GB heap each. Our code is completely unchanged
>> except as required to adapt to the new Giraph APIs. Our vertex, edge, and
>> message data are completely unchanged. On smaller jobs, that work, the
>> aggregate heap usage high-water mark seems about the same as before, but
>> the "committed heap" seems to run higher. I can't even make it work on a
>> cluster of 12. In that case I get one map task that seems to end up with
>> nearly twice as many messages as most of the others so it runs out of
>> memory anyway. It only takes one to fail the job. Am I missing something
>> here? Should I be configuring my new Giraph in some way I didn't used to
>> need to with the old one?
>>
>>
>

Mime
View raw message