hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 심탁길 <1004...@nhncorp.com>
Subject Jobtracker is out of memory with 100,000 dummy map tasks job
Date Fri, 26 Sep 2008 06:26:37 GMT
 
Hi all
 
Recently I tried 100,000 dummy map tasks job on the 100ea node(2GB, Dual Core, 64Bit machine,
Version: 0.16.4) cluster
 
Map task does nothing but sleeping one minute
 
I found that Jobtracker(1GB Heap) consumes about 650MB of heap memory when the job is 50%
done.
 
After all, the job failed at the 90% of progress because Jobtracker hanged up(?) due to out
of memory.
 
how do you handle this kind of issue? 
 
another related issue:
 
while the above job was being processed, I clicked on the "Pending" on jobdatails.jsp of web
UI
 
then, Jobtracker consumed 100% of CPU. and 100% CPU status lasted a couple of minutes


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message