hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Jobtracker is out of memory with 100,000 dummy map tasks job
Date Fri, 26 Sep 2008 08:11:05 GMT

On Sep 25, 2008, at 11:26 PM, 심탁길 wrote:

>
> Hi all
>
> Recently I tried 100,000 dummy map tasks job on the 100ea node(2GB,  
> Dual Core, 64Bit machine, Version: 0.16.4) cluster
>
> Map task does nothing but sleeping one minute
>
> I found that Jobtracker(1GB Heap) consumes about 650MB of heap  
> memory when the job is 50% done.
>
> After all, the job failed at the 90% of progress because Jobtracker  
> hanged up(?) due to out of memory.
>

Funny that you bring this up today... I just now concluded that we  
need to port
https://issues.apache.org/jira/browse/HADOOP-3670 to 0.17 branch to  
fix a sluggish JobTracker who was spending all his time doing GC.

Please upgrade to 0.18 if you can, immdediately! *smile*

Arun

> how do you handle this kind of issue?
>
> another related issue:
>
> while the above job was being processed, I clicked on the "Pending"  
> on jobdatails.jsp of web UI
>
> then, Jobtracker consumed 100% of CPU. and 100% CPU status lasted a  
> couple of minutes
>


Mime
View raw message