Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 34439 invoked from network); 7 Jun 2007 04:31:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 7 Jun 2007 04:31:50 -0000 Received: (qmail 75732 invoked by uid 500); 7 Jun 2007 04:31:54 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 75433 invoked by uid 500); 7 Jun 2007 04:31:53 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 75420 invoked by uid 99); 7 Jun 2007 04:31:53 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jun 2007 21:31:53 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jun 2007 21:31:48 -0700 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 5BC857141A4 for ; Wed, 6 Jun 2007 21:31:28 -0700 (PDT) Message-ID: <23299311.1181190688372.JavaMail.jira@brutus> Date: Wed, 6 Jun 2007 21:31:28 -0700 (PDT) From: "Doug Cutting (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Commented: (HADOOP-1158) JobTracker should collect statistics of failed map output fetches, and take decisions to reexecute map tasks and/or restart the (possibly faulty) Jetty server on the TaskTracker In-Reply-To: <7318675.1174848452133.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502216 ] Doug Cutting commented on HADOOP-1158: -------------------------------------- Putting all status into a single object sounds like a good approach. Perhaps the method should be renamed 'status' rather than 'progress'? > JobTracker should collect statistics of failed map output fetches, and take decisions to reexecute map tasks and/or restart the (possibly faulty) Jetty server on the TaskTracker > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: HADOOP-1158 > URL: https://issues.apache.org/jira/browse/HADOOP-1158 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Affects Versions: 0.12.2 > Reporter: Devaraj Das > Assignee: Arun C Murthy > > The JobTracker should keep a track (with feedback from Reducers) of how many times a fetch for a particular map output failed. If this exceeds a certain threshold, then that map should be declared as lost, and should be reexecuted elsewhere. Based on the number of such complaints from Reducers, the JobTracker can blacklist the TaskTracker. This will make the framework reliable - it will take care of (faulty) TaskTrackers that sometimes always fail to serve up map outputs (for which exceptions are not properly raised/handled, for e.g., if the exception/problem happens in the Jetty server). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.