hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Gao <steve....@yahoo.com>
Subject Re: [Interesting] One reducer randomly hangs on getting 0 mapper output
Date Fri, 10 Apr 2009 16:49:26 GMT
Does anybody have a clue? Thanks lot.

--- On Thu, 4/9/09, Steve Gao <steve.gao@yahoo.com> wrote:

From: Steve Gao <steve.gao@yahoo.com>
Subject: [Interesting] One reducer randomly hangs on getting 0 mapper output
To: core-user@hadoop.apache.org
Date: Thursday, April 9, 2009, 6:04 PM


I have hadoop jobs with the last 1 reducer randomly hangs on getting 0 mapper output. By randomly
I mean the job sometimes works correctly, sometimes their last 1 reducer keeps reading map
output but always gets 0 data. It would hang up to 100 hours for getting 0 data until I kill
it. After I kill and re-run it, it could run correctly. The hung reducer could happen on any
machine of my cluster.

I attach the tail of the problematic reducer's log here. Does anybody have a hint what happened?

syslog logs

2009-04-09 21:57:46,445 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Need 15 map output(s)
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Got 0 known map output location(s); scheduling...
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)

2009-04-09 21:57:51,453 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Need 15 map output(s)
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Got 0 known map output location(s); scheduling...
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)


... (forever)



      


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message