Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1536919889.1244493847817.JavaMail.jira@brutus>
Date: Mon, 8 Jun 2009 13:44:07 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5985) A single slow (but not dead) map
 TaskTracker impedes MapReduce progress
In-Reply-To: <901992913.1244247967376.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717401#action_12717401 ] 

Aaron Kimball commented on HADOOP-5985:
---------------------------------------

I was under the impression that if a map task died before delivering output to all of the reducers, then after the map task is re-executed elsewhere, all reducers roll back and re-pull from the newer version of the mapper. I could be mistaken though?

Because if that's the case, then we just need to run another round of speculative execution during the final shufflings, even if the map tasks themselves were marked as "complete," and modify the reducers to try to pull from a list of eligible mappers instead of just a single node.

> A single slow (but not dead) map TaskTracker impedes MapReduce progress
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-5985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5985
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.18.3
>            Reporter: Aaron Kimball
>
> We see cases where there may be a large number of mapper nodes running many tasks (e.g., a thousand). The reducers will pull 980 of the map task intermediate files down, but will be unable to retrieve the final intermediate shards from the last node. The TaskTracker on that node returns data to reducers either slowly or not at all, but its heartbeat messages make it back to the JobTracker -- so the JobTracker doesn't mark the tasks as failed. Manually stopping the offending TaskTracker works to migrate the tasks to other nodes, where the shuffling process finishes very quickly. Left on its own, it can take hours to unjam itself otherwise.
> We need a mechanism for reducers to provide feedback to the JobTracker that one of the mapper nodes should be regarded as lost.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.