Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 35989 invoked from network); 13 Feb 2009 03:57:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Feb 2009 03:57:16 -0000 Received: (qmail 31011 invoked by uid 500); 13 Feb 2009 03:57:16 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 30970 invoked by uid 500); 13 Feb 2009 03:57:16 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 30961 invoked by uid 99); 13 Feb 2009 03:57:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2009 19:57:16 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2009 03:57:15 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 1D7E12388A9A; Fri, 13 Feb 2009 03:56:55 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r743973 - in /hadoop/core/branches/branch-0.20: ./ CHANGES.txt src/mapred/org/apache/hadoop/mapred/TaskInProgress.java src/test/org/apache/hadoop/mapred/MiniMRCluster.java src/test/org/apache/hadoop/mapred/TestLostTracker.java Date: Fri, 13 Feb 2009 03:56:54 -0000 To: core-commits@hadoop.apache.org From: ddas@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20090213035655.1D7E12388A9A@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: ddas Date: Fri Feb 13 03:56:54 2009 New Revision: 743973 URL: http://svn.apache.org/viewvc?rev=743973&view=rev Log: Merge -r 743815:743816 from trunk onto 0.20 branch. Fixes HADOOP-5067. Added: hadoop/core/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestLostTracker.java - copied unchanged from r743816, hadoop/core/trunk/src/test/org/apache/hadoop/mapred/TestLostTracker.java Modified: hadoop/core/branches/branch-0.20/ (props changed) hadoop/core/branches/branch-0.20/CHANGES.txt (contents, props changed) hadoop/core/branches/branch-0.20/src/mapred/org/apache/hadoop/mapred/TaskInProgress.java hadoop/core/branches/branch-0.20/src/test/org/apache/hadoop/mapred/MiniMRCluster.java Propchange: hadoop/core/branches/branch-0.20/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Feb 13 03:56:54 2009 @@ -1,2 +1,2 @@ /hadoop/core/branches/branch-0.19:713112 -/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743892 +/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892 Modified: hadoop/core/branches/branch-0.20/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.20/CHANGES.txt?rev=743973&r1=743972&r2=743973&view=diff ============================================================================== --- hadoop/core/branches/branch-0.20/CHANGES.txt (original) +++ hadoop/core/branches/branch-0.20/CHANGES.txt Fri Feb 13 03:56:54 2009 @@ -687,6 +687,9 @@ HADOOP-5166. Fix JobTracker restart to work when ACLs are configured for the JobTracker. (Amar Kamat via yhemanth). + HADOOP-5067. Fixes TaskInProgress.java to keep track of count of failed and + killed tasks correctly. (Amareshwari Sriramadasu via ddas) + Release 0.19.0 - 2008-11-18 INCOMPATIBLE CHANGES Propchange: hadoop/core/branches/branch-0.20/CHANGES.txt ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Feb 13 03:56:54 2009 @@ -1,3 +1,3 @@ /hadoop/core/branches/branch-0.18/CHANGES.txt:727226 /hadoop/core/branches/branch-0.19/CHANGES.txt:713112 -/hadoop/core/trunk/CHANGES.txt:727001,727117,727191,727212,727228,727255,727869,728187,729052,729987,732385,732572,732777,732838,732869,733887,734870,734916,735082,736426,738602,738697,739416,740077,740157,741703,741762,743296,743745,743892 +/hadoop/core/trunk/CHANGES.txt:727001,727117,727191,727212,727228,727255,727869,728187,729052,729987,732385,732572,732777,732838,732869,733887,734870,734916,735082,736426,738602,738697,739416,740077,740157,741703,741762,743296,743745,743816,743892 Modified: hadoop/core/branches/branch-0.20/src/mapred/org/apache/hadoop/mapred/TaskInProgress.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.20/src/mapred/org/apache/hadoop/mapred/TaskInProgress.java?rev=743973&r1=743972&r2=743973&view=diff ============================================================================== --- hadoop/core/branches/branch-0.20/src/mapred/org/apache/hadoop/mapred/TaskInProgress.java (original) +++ hadoop/core/branches/branch-0.20/src/mapred/org/apache/hadoop/mapred/TaskInProgress.java Fri Feb 13 03:56:54 2009 @@ -94,6 +94,8 @@ // Map from task Id -> TaskTracker Id, contains tasks that are // currently runnings private TreeMap activeTasks = new TreeMap(); + // All attempt Ids of this TIP + private TreeSet tasks = new TreeSet(); private JobConf conf; private Map> taskDiagnosticData = new TreeMap>(); @@ -602,9 +604,7 @@ } } - // Note that there can be failures of tasks that are hosted on a machine - // that has not yet registered with restarted jobtracker - boolean isPresent = this.activeTasks.remove(taskid) != null; + this.activeTasks.remove(taskid); // Since we do not fail completed reduces (whose outputs go to hdfs), we // should note this failure only for completed maps, only if this taskid; @@ -618,8 +618,10 @@ resetSuccessfulTaskid(); } + // Note that there can be failures of tasks that are hosted on a machine + // that has not yet registered with restarted jobtracker // recalculate the counts only if its a genuine failure - if (isPresent) { + if (tasks.contains(taskid)) { if (taskState == TaskStatus.State.FAILED) { numTaskFailures++; machinesWhereFailed.add(trackerHostName); @@ -924,6 +926,7 @@ } activeTasks.put(taskid, taskTracker); + tasks.add(taskid); // Ask JobTracker to note that the task exists jobtracker.createTaskEntry(taskid, taskTracker, this); Modified: hadoop/core/branches/branch-0.20/src/test/org/apache/hadoop/mapred/MiniMRCluster.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.20/src/test/org/apache/hadoop/mapred/MiniMRCluster.java?rev=743973&r1=743972&r2=743973&view=diff ============================================================================== --- hadoop/core/branches/branch-0.20/src/test/org/apache/hadoop/mapred/MiniMRCluster.java (original) +++ hadoop/core/branches/branch-0.20/src/test/org/apache/hadoop/mapred/MiniMRCluster.java Fri Feb 13 03:56:54 2009 @@ -235,6 +235,9 @@ return jobTracker; } + TaskTrackerRunner getTaskTrackerRunner(int id) { + return taskTrackerList.get(id); + } /** * Get the number of task trackers in the cluster */