Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 72432 invoked from network); 3 Jun 2008 03:57:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 3 Jun 2008 03:57:41 -0000 Received: (qmail 2032 invoked by uid 500); 3 Jun 2008 03:57:44 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 2010 invoked by uid 500); 3 Jun 2008 03:57:44 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 2001 invoked by uid 99); 3 Jun 2008 03:57:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jun 2008 20:57:43 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 03:56:56 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 1252A2388A06; Mon, 2 Jun 2008 20:57:18 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r662631 - in /hadoop/core/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/MapTask.java Date: Tue, 03 Jun 2008 03:57:17 -0000 To: core-commits@hadoop.apache.org From: cdouglas@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20080603035718.1252A2388A06@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: cdouglas Date: Mon Jun 2 20:57:17 2008 New Revision: 662631 URL: http://svn.apache.org/viewvc?rev=662631&view=rev Log: HADOOP-3475. Fix MapTask to correctly size the accounting allocation of io.sort.mb. Modified: hadoop/core/trunk/CHANGES.txt hadoop/core/trunk/src/java/org/apache/hadoop/mapred/MapTask.java Modified: hadoop/core/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=662631&r1=662630&r2=662631&view=diff ============================================================================== --- hadoop/core/trunk/CHANGES.txt (original) +++ hadoop/core/trunk/CHANGES.txt Mon Jun 2 20:57:17 2008 @@ -388,6 +388,9 @@ HADOOP-3471. Fix spurious errors from TestIndexedSort and add additional logging to let failures be reproducible. (cdouglas) + HADOOP-3475. Fix MapTask to correctly size the accounting allocation of + io.sort.mb. (cdouglas) + Release 0.17.0 - 2008-05-18 INCOMPATIBLE CHANGES Modified: hadoop/core/trunk/src/java/org/apache/hadoop/mapred/MapTask.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/mapred/MapTask.java?rev=662631&r1=662630&r2=662631&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/mapred/MapTask.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/mapred/MapTask.java Mon Jun 2 20:57:17 2008 @@ -324,6 +324,8 @@ private static final int KEYSTART = 1; // key offset in acct private static final int VALSTART = 2; // val offset in acct private static final int ACCTSIZE = 3; // total #fields in acct + private static final int RECSIZE = + (ACCTSIZE + 1) * 4; // acct bytes per record // spill accounting private volatile int numSpills = 0; @@ -367,12 +369,12 @@ // buffers and accounting int maxMemUsage = sortmb << 20; int recordCapacity = (int)(maxMemUsage * recper); - recordCapacity += (recordCapacity >>> 2) % 4; + recordCapacity -= recordCapacity % RECSIZE; kvbuffer = new byte[maxMemUsage - recordCapacity]; bufvoid = kvbuffer.length; - int kvcapacity = recordCapacity >>> 2; - kvoffsets = new int[kvcapacity]; - kvindices = new int[recordCapacity - kvcapacity]; + recordCapacity /= RECSIZE; + kvoffsets = new int[recordCapacity]; + kvindices = new int[recordCapacity * ACCTSIZE]; softBufferLimit = (int)(kvbuffer.length * spillper); softRecordLimit = (int)(kvoffsets.length * spillper); // k/v serialization