hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5985) A single slow (but not dead) map TaskTracker impedes MapReduce progress
Date Sat, 06 Jun 2009 00:28:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716808#action_12716808
] 

Aaron Kimball commented on HADOOP-5985:
---------------------------------------


Here's a first stab at an algorithm to fix this:

*  Let's define the "home stretch" of shuffling, from the perspective of a reducer, as the
last X% of the map shards it needs to fetch. Maybe that's 3% or 4%.
* If during the home stretch, a map task enters the penalty box and stays there for some number
of seconds (let's say 30), the reducer can send a message back to the JobTracker voting that
the task be marked as "failed for being too slow."
* The JobTracker requires a quorum of still-shuffling reducers to vote this way. The quorum
might be, say, 10%. This way if only two reducers are still waiting, either one of them is
a quorum by itself. But if 40 reducers are shuffling, it will take the consensus of a few
reducers that a mapper is indeed too slow.
* If enough votes are given, the JobTracker blacklists a TaskTracker from the job, marks all
its map task attempts as failed, and restarts all the map tasks from that node, on other nodes.


> A single slow (but not dead) map TaskTracker impedes MapReduce progress
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-5985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5985
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.18.3
>            Reporter: Aaron Kimball
>
> We see cases where there may be a large number of mapper nodes running many tasks (e.g.,
a thousand). The reducers will pull 980 of the map task intermediate files down, but will
be unable to retrieve the final intermediate shards from the last node. The TaskTracker on
that node returns data to reducers either slowly or not at all, but its heartbeat messages
make it back to the JobTracker -- so the JobTracker doesn't mark the tasks as failed. Manually
stopping the offending TaskTracker works to migrate the tasks to other nodes, where the shuffling
process finishes very quickly. Left on its own, it can take hours to unjam itself otherwise.
> We need a mechanism for reducers to provide feedback to the JobTracker that one of the
mapper nodes should be regarded as lost.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message