Return-Path: Delivered-To: apmail-lucene-hadoop-commits-archive@locus.apache.org Received: (qmail 31410 invoked from network); 21 Apr 2007 18:56:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 21 Apr 2007 18:56:00 -0000 Received: (qmail 36137 invoked by uid 500); 21 Apr 2007 18:56:06 -0000 Delivered-To: apmail-lucene-hadoop-commits-archive@lucene.apache.org Received: (qmail 36051 invoked by uid 500); 21 Apr 2007 18:56:06 -0000 Mailing-List: contact hadoop-commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-commits@lucene.apache.org Received: (qmail 36039 invoked by uid 99); 21 Apr 2007 18:56:06 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Apr 2007 11:56:06 -0700 X-ASF-Spam-Status: No, hits=-99.5 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO eris.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Apr 2007 11:55:59 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 4761C1A9838; Sat, 21 Apr 2007 11:55:39 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r531084 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/ReduceTask.java Date: Sat, 21 Apr 2007 18:55:39 -0000 To: hadoop-commits@lucene.apache.org From: tomwhite@apache.org X-Mailer: svnmailer-1.1.0 Message-Id: <20070421185539.4761C1A9838@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: tomwhite Date: Sat Apr 21 11:55:38 2007 New Revision: 531084 URL: http://svn.apache.org/viewvc?view=rev&rev=531084 Log: HADOOP-1152. Fix race condition in MapOutputCopier.copyOutput file rename causing possible reduce task hang. Contributed by Tahir Hashmi. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?view=diff&rev=531084&r1=531083&r2=531084 ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Sat Apr 21 11:55:38 2007 @@ -234,6 +234,10 @@ 70. HADOOP-1275. Fix misspelled job notification property in hadoop-default.xml. (Alejandro Abdelnur via tomwhite) +71. HADOOP-1152. Fix race condition in MapOutputCopier.copyOutput file + rename causing possible reduce task hang. + (Tahir Hashmi via tomwhite) + Release 0.12.3 - 2007-04-06 Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java?view=diff&rev=531084&r1=531083&r2=531084 ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java Sat Apr 21 11:55:38 2007 @@ -698,14 +698,17 @@ fs.delete(tmpFilename); return CopyResult.OBSOLETE; } + + bytes = fs.getLength(tmpFilename); // if we can't rename the file, something is broken (and IOException // will be thrown). if (!fs.rename(tmpFilename, finalFilename)) { fs.delete(tmpFilename); + bytes = -1; throw new IOException("failure to rename map output " + tmpFilename); } - bytes = fs.getLength(finalFilename); + LOG.info(reduceId + " done copying " + loc.getMapTaskId() + " output from " + loc.getHost() + "."); //Create a thread to do merges. Synchronize access/update to