hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Gao <steve....@yahoo.com>
Subject [Interesting] One reducer randomly hangs on getting 0 mapper output
Date Thu, 09 Apr 2009 22:04:53 GMT

I have hadoop jobs with the last 1 reducer randomly hangs on getting 0 mapper output. By randomly
I mean the job sometimes works correctly, sometimes their last 1 reducer keeps reading map
output but always gets 0 data. It would hang up to 100 hours for getting 0 data until I kill
it. After I kill and re-run it, it could run correctly. The hung reducer could happen on any
machine of my cluster.

I attach the tail of the problematic reducer's log here. Does anybody have a hint what happened?

syslog logs

2009-04-09 21:57:46,445 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Need 15 map output(s)
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Got 0 known map output location(s); scheduling...
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)

2009-04-09 21:57:51,453 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Need 15 map output(s)
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Got 0 known map output location(s); scheduling...
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)


... (forever)



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message