hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Gao <steve....@yahoo.com>
Subject Re: [Interesting] One reducer randomly hangs on getting 0 mapper output
Date Thu, 09 Apr 2009 22:19:17 GMT
I am using 0.17.0 . 
I think the problem is basically because reducer falls in a infinite loop to get mapper output,
when mapper is somehow not available/dead . Doesn't hadoop have a solution?

--- On Thu, 4/9/09, Steve Gao <steve.gao@yahoo.com> wrote:

From: Steve Gao <steve.gao@yahoo.com>
Subject: [Interesting] One reducer randomly hangs on getting 0 mapper output
To: core-user@hadoop.apache.org
Date: Thursday, April 9, 2009, 6:04 PM


I have hadoop jobs with the last 1 reducer randomly hangs on getting 0 mapper output. By randomly
I mean the job sometimes works correctly, sometimes their last 1 reducer keeps reading map
output but always gets 0 data. It would hang up to 100 hours for getting 0 data until I kill
it. After I kill and re-run it, it could run correctly. The hung reducer could happen on any
machine of my cluster.

I attach the tail of the problematic reducer's log here. Does anybody have a hint what happened?

syslog logs

2009-04-09 21:57:46,445 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Need 15 map output(s)
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Got 0 known map output location(s); scheduling...
2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)

2009-04-09 21:57:51,453 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Need 15 map output(s)
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Got 0 known map output location(s); scheduling...
2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0
Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)


... (forever)



      


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message