Return-Path: Delivered-To: apmail-lucene-hadoop-commits-archive@locus.apache.org Received: (qmail 36774 invoked from network); 9 Jan 2007 21:37:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 9 Jan 2007 21:37:25 -0000 Received: (qmail 26110 invoked by uid 500); 9 Jan 2007 21:37:32 -0000 Delivered-To: apmail-lucene-hadoop-commits-archive@lucene.apache.org Received: (qmail 26031 invoked by uid 500); 9 Jan 2007 21:37:32 -0000 Mailing-List: contact hadoop-commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-commits@lucene.apache.org Received: (qmail 26022 invoked by uid 99); 9 Jan 2007 21:37:32 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jan 2007 13:37:32 -0800 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO eris.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jan 2007 13:37:24 -0800 Received: by eris.apache.org (Postfix, from userid 65534) id F07771A981C; Tue, 9 Jan 2007 13:36:24 -0800 (PST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r494602 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/MapTask.java Date: Tue, 09 Jan 2007 21:36:24 -0000 To: hadoop-commits@lucene.apache.org From: cutting@apache.org X-Mailer: svnmailer-1.1.0 Message-Id: <20070109213624.F07771A981C@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: cutting Date: Tue Jan 9 13:36:24 2007 New Revision: 494602 URL: http://svn.apache.org/viewvc?view=rev&rev=494602 Log: HADOOP-868. Decrease the number of files opened during map. Contributed by Devaraj. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?view=diff&rev=494602&r1=494601&r2=494602 ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue Jan 9 13:36:24 2007 @@ -23,6 +23,9 @@ 7. HADOOP-871. Fix a bug in bin/hadoop setting JAVA_LIBRARY_PATH. (Arun C Murthy via cutting) + 8. HADOOP-868. Decrease the number of open files during map, + respecting io.sort.factor. (Devaraj Das via cutting) + Release 0.10.0 - 2007-01-05 Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java?view=diff&rev=494602&r1=494601&r2=494602 ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java Tue Jan 9 13:36:24 2007 @@ -467,26 +467,24 @@ { Path [] filename = new Path[numSpills]; Path [] indexFileName = new Path[numSpills]; - FSDataInputStream in[] = new FSDataInputStream[numSpills]; - FSDataInputStream indexIn[] = new FSDataInputStream[numSpills]; for(int i = 0; i < numSpills; i++) { filename[i] = mapOutputFile.getSpillFile(getTaskId(), i); - in[i] = localFs.open(filename[i]); indexFileName[i] = mapOutputFile.getSpillIndexFile(getTaskId(), i); - indexIn[i] = localFs.open(indexFileName[i]); } //create a sorter object as we need access to the SegmentDescriptor //class and merge methods Sorter sorter = new Sorter(localFs, keyClass, valClass, job); - sorter.setFactor(numSpills); for (int parts = 0; parts < partitions; parts++){ List segmentList = new ArrayList(numSpills); for(int i = 0; i < numSpills; i++) { - long segmentOffset = indexIn[i].readLong(); - long segmentLength = indexIn[i].readLong(); + FSDataInputStream indexIn = localFs.open(indexFileName[i]); + indexIn.seek(parts * 16); + long segmentOffset = indexIn.readLong(); + long segmentLength = indexIn.readLong(); + indexIn.close(); SegmentDescriptor s = sorter.new SegmentDescriptor(segmentOffset, segmentLength, filename[i]); s.preserveInput(true); @@ -513,8 +511,8 @@ finalIndexOut.close(); //cleanup for(int i = 0; i < numSpills; i++) { - in[i].close(); localFs.delete(filename[i]); - indexIn[i].close(); localFs.delete(indexFileName[i]); + localFs.delete(filename[i]); + localFs.delete(indexFileName[i]); } } }