Return-Path: Delivered-To: apmail-hadoop-core-dev-archive@www.apache.org Received: (qmail 54747 invoked from network); 22 Jun 2009 02:46:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 22 Jun 2009 02:46:20 -0000 Received: (qmail 45234 invoked by uid 500); 22 Jun 2009 02:46:30 -0000 Delivered-To: apmail-hadoop-core-dev-archive@hadoop.apache.org Received: (qmail 45144 invoked by uid 500); 22 Jun 2009 02:46:30 -0000 Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-dev@hadoop.apache.org Received: (qmail 44938 invoked by uid 99); 22 Jun 2009 02:46:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 02:46:29 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 02:46:27 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id B75BB234C1F0 for ; Sun, 21 Jun 2009 19:46:07 -0700 (PDT) Message-ID: <711073515.1245638767750.JavaMail.jira@brutus> Date: Sun, 21 Jun 2009 19:46:07 -0700 (PDT) From: "Arun C Murthy (JIRA)" To: core-dev@hadoop.apache.org Subject: [jira] Updated: (HADOOP-5985) A single slow (but not dead) map TaskTracker impedes MapReduce progress In-Reply-To: <901992913.1244247967376.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HADOOP-5985: ---------------------------------- Component/s: mapred > A single slow (but not dead) map TaskTracker impedes MapReduce progress > ----------------------------------------------------------------------- > > Key: HADOOP-5985 > URL: https://issues.apache.org/jira/browse/HADOOP-5985 > Project: Hadoop Common > Issue Type: Bug > Components: mapred > Affects Versions: 0.18.3 > Reporter: Aaron Kimball > > We see cases where there may be a large number of mapper nodes running many tasks (e.g., a thousand). The reducers will pull 980 of the map task intermediate files down, but will be unable to retrieve the final intermediate shards from the last node. The TaskTracker on that node returns data to reducers either slowly or not at all, but its heartbeat messages make it back to the JobTracker -- so the JobTracker doesn't mark the tasks as failed. Manually stopping the offending TaskTracker works to migrate the tasks to other nodes, where the shuffling process finishes very quickly. Left on its own, it can take hours to unjam itself otherwise. > We need a mechanism for reducers to provide feedback to the JobTracker that one of the mapper nodes should be regarded as lost. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.