hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: The reduce copier failed
Date Thu, 25 Sep 2008 21:32:00 GMT

On Sep 25, 2008, at 2:26 PM, Joe Shaw wrote:

> Hi,
>
> I'm trying to build an index using the "index" contrib in Hadoop
> 0.18.0, but the reduce tasks are consistently failing.
>

What did the logs for the task-attempt  
'attempt_200809180916_0027_r_000007_2' look like? Did the TIP/Job  
succeed?

Arun

> In the output from the "hadoop jar" command, I see messages like this:
>
> 08/09/25 14:12:11 INFO mapred.JobClient:  map 27% reduce 4%
> 08/09/25 14:12:23 INFO mapred.JobClient: Task Id :
> attempt_200809180916_0027_r_000007_2, Status : FAILED
> java.io.IOException: attempt_200809180916_0027_r_000007_2The reduce
> copier failed
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java: 
> 2209)
>
> and eventually failing.
>
> The output from "hadoop job -history" gives me:
>
> Task Summary
> ============================
> Kind    Total   Successful      Failed  Killed  StartTime        
> FinishTime
>
> Map     57      57              0       0       25-Sep-2008 14:03:07
> 25-Sep-2008 14:13:17 (10mins, 9sec)
> Reduce  4       0               4       0       25-Sep-2008 14:03:14
> 25-Sep-2008 14:13:21 (10mins, 7sec)
> ============================
>
> and
>
> FAILED REDUCE task list for job_200809180916_0027
> TaskId          StartTime       FinishTime      Error
> ====================================================
> task_200809180916_0027_r_000007 25-Sep-2008 14:03:14    25-Sep-2008
> 14:13:21 (10mins, 7sec)
>
> Grepping in the logs for that task, I see this consistently on the  
> TaskTrackers:
>
> hadoop-jshaw-tasktracker-ars1dev3.log:2008-09-25 14:09:08,685 INFO
> org.apache.hadoop.mapred.TaskTracker:
> attempt_200809180916_0027_r_000007_1 0.016147636% reduce > copy (14 of
> 289 at 8.37 MB/s) >
> hadoop-jshaw-tasktracker-ars1dev3.log:2008-09-25 14:09:11,904 INFO
> org.apache.hadoop.mapred.TaskTracker:
> attempt_200809180916_0027_r_000007_1 0.018454442% reduce > copy (16 of
> 289 at 7.85 MB/s) >
> hadoop-jshaw-tasktracker-ars1dev3.log:2008-09-25 14:09:17,337 INFO
> org.apache.hadoop.mapred.TaskRunner:
> attempt_200809180916_0027_r_000007_1 done; removing files.
>
> As you can see, it apparently is unable to copy the data, but it gives
> me absolutely no idea why.  The JobTracker logs also give me no useful
> information.
>
> Anybody have an idea what's going on, or how I might go about  
> debugging this?
>
> Thanks,
> Joe


Mime
View raw message