giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Out of memory with giraph-release-1.0.0-RC3, used to work on old Giraph
Date Wed, 28 Aug 2013 23:57:20 GMT
Try dumping a histogram of memory usage from a running JVM and see where 
the memory is going.  I can't think of anything in particular that 
changed...

On 8/28/13 4:39 PM, Jeff Peters wrote:
>
> I am tasked with updating our ancient (circa 7/10/2012) Giraph to 
> giraph-release-1.0.0-RC3. Most jobs run fine but our largest job now 
> runs out of memory using the same AWS elastic-mapreduce configuration 
> we have always used. I have never tried to configure either Giraph or 
> the AWS Hadoop. We build for Hadoop 1.0.2 because that's closest to 
> the 1.0.3 AWS provides us. The 8 X m2.4xlarge cluster we use seems to 
> provide 8*14=112 map tasks fitted out with 2GB heap each. Our code is 
> completely unchanged except as required to adapt to the new Giraph 
> APIs. Our vertex, edge, and message data are completely unchanged. On 
> smaller jobs, that work, the aggregate heap usage high-water mark 
> seems about the same as before, but the "committed heap" seems to run 
> higher. I can't even make it work on a cluster of 12. In that case I 
> get one map task that seems to end up with nearly twice as many 
> messages as most of the others so it runs out of memory anyway. It 
> only takes one to fail the job. Am I missing something here? Should I 
> be configuring my new Giraph in some way I didn't used to need to with 
> the old one?
>


Mime
View raw message