hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thushara Wijeratna <thu...@gmail.com>
Subject child task choking on large (19K) input line
Date Fri, 29 May 2009 19:53:58 GMT
has anyone seen this? basically a child task is killing itself, as a ping
with the parent didn't quite work - the reply from the parent was
unexpected.

hadoop version: 0.19.0
userlogs on slave node:

2009-05-29 13:57:33,551 WARN org.apache.hadoop.mapred.TaskRunner: Parent
died.  Exiting attempt_200905281652_0013_m_000006_1
[root@domU-12-31-38-01-7C-92 attempt_200905281652_0013_m_000006_1]#

tellingly, the last input line processed right before this WARN is 19K. (i
log the full input line in the map function for debugging)

output on map-reduce task:

Task attempt_200905281652_0013_m_000006_2 failed to report status for 600
seconds. Killing!
09/05/29 14:08:01 INFO mapred.JobClient:  map 99% reduce 32%
09/05/29 14:18:05 INFO mapred.JobClient:  map 98% reduce 32%
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217)
    at
com.adxpose.data.mr.DailyHeatmapAggregator.run(DailyHeatmapAggregator.java:547)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at
com.adxpose.data.mr.DailyHeatmapAggregator.main(DailyHeatmapAggregator.java:553)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
    at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

i believe this is the code that kills the child:

in org.apache.hadoop.mapred.Task

protected void startCommunicationThread(final TaskUmbilicalProtocol
umbilical) {

....

              if (sendProgress) {
                // we need to send progress update
                updateCounters();
                taskStatus.statusUpdate(getState(),
                                        taskProgress.get(),
                                        taskProgress.toString(),
                                        counters);
                taskFound = umbilical.statusUpdate(taskId, taskStatus);
                taskStatus.clearStatus();
              }
              else {
                // send ping
                taskFound = umbilical.ping(taskId);
              }

              // if Task Tracker is not aware of our task ID (probably
because it died and
              // came back up), kill ourselves
              if (!taskFound) {
                LOG.warn("Parent died.  Exiting "+taskId);
                System.exit(66);
              }

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message