hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johan Oskarsson <jo...@oskarsson.nu>
Subject TaskTracker out of memoy error
Date Wed, 28 Feb 2007 14:15:58 GMT
Hi.

I've run into problems with the tasktrackers dying when I run a fairly 
big job. The input data is roughly 290 gb and the tasktrackers die 
because of out of memory errors.
This job is run on a 19 node hadoop 0.11.2 cluster with the tasktracker 
memory set to 1200 mb. I could increase this further but at
some point the machines will obviously hit the memory limit.

Unfortunately the input data is divided into 50-100 mb files accompanied 
by very small files (a few kb) that are added later on as we can't 
append to the big files.
The number of map tasks for this job is 29638, could this have anything 
to do with the problem? The job dies about 70% into the map phase.

Exception found in the log of one of the task trackers:

2007-02-27 23:59:58,800 INFO org.apache.hadoop.mapred.TaskTracker: Error 
cleaning up task runner: java.lang.NullPointerException
        at 
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:1192)
        at 
org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:371)
        at 
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:360)
        at 
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:828)
        at 
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:511)
        at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:857)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1499)

2007-02-27 23:59:58,800 INFO org.apache.hadoop.mapred.TaskTracker: 
java.lang.OutOfMemoryError: Java heap space


Does anyone have a good idea why this happens and what I can do to solve it?

/Johan

Mime
View raw message