hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: JobTracker memory usage peaks once a day and OOM sometimes
Date Tue, 08 Feb 2011 21:00:53 GMT

On Feb 8, 2011, at 12:56 PM, Arun C Murthy wrote:

> As Allen mentioned it's really hard for anyone here to help you with
> CDH. You'll have to ask their user lists.
>
> OTOH, there are significant enhancements we've done to the JobTracker
> in 0.21 and 0.22... the problem is that there isn't an Apache release
> yet.
>
> the JT we run at Y! has a *huge* number of fixes for memory and
> performance (>100% improvement) which we've started putting into a
> 0.20.100 branch - you might be interested in using that if you want a
> packaged release: http://people.apache.org/~acmurthy/hadoop-0.20.100-rc0/
>

I should point out that this isn't an 'Apache Release', just something  
for you to try if you are interested.

Arun

> Arun
>
> On Feb 8, 2011, at 8:59 AM, Maxim Zizin wrote:
>
>> Hi all,
>>
>> We monitor JT, NN and SNN memory usage and observe the following
>> behavior in our Hadoop cluster. JT's heap size is set to 2000m.
>> About 18
>> hours a day it uses ~1GB but every day roughly at the minute it was
>> started its used memory increases to ~1.5GB and then decreases back  
>> to
>> ~1GB in about 6 hours. Sometimes this takes a bit more than 6 hours,
>> sometimes a bit less. I was wondering whether anyone here knows what
>> JT
>> does once a day that makes it use 1.5 times more memory than  
>> normally.
>>
>> We're so interested in JT memory usage because during last two weeks
>> we
>> twice had JT getting out of heap space. Both times right after those
>> daily used memory peaks when it was going down from 1.5GB to 1GB it
>> started increasing again until got stuck at ~2.2GB. After that it
>> becomes unresponsive and we have to restart it.
>>
>> We're using Cloudera's CDH2 version 0.20.1+169.113.
>>
>> -- 
>> Regards, Max
>


Mime
View raw message