Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 73455 invoked from network); 19 Jun 2008 12:52:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 19 Jun 2008 12:52:21 -0000 Received: (qmail 649 invoked by uid 500); 19 Jun 2008 12:52:23 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 619 invoked by uid 500); 19 Jun 2008 12:52:23 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 610 invoked by uid 99); 19 Jun 2008 12:52:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Jun 2008 05:52:23 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Jun 2008 12:51:41 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 97BA723889C2; Thu, 19 Jun 2008 05:52:00 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r669474 - in /hadoop/core/branches/branch-0.18: CHANGES.txt src/core/org/apache/hadoop/fs/HarFileSystem.java src/mapred/org/apache/hadoop/mapred/FileInputFormat.java Date: Thu, 19 Jun 2008 12:52:00 -0000 To: core-commits@hadoop.apache.org From: ddas@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20080619125200.97BA723889C2@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: ddas Date: Thu Jun 19 05:51:59 2008 New Revision: 669474 URL: http://svn.apache.org/viewvc?rev=669474&view=rev Log: Merge -r 669471:669472 from trunk onto 0.18 branch. Fixes HADOOP-3580. Modified: hadoop/core/branches/branch-0.18/CHANGES.txt hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/HarFileSystem.java hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/FileInputFormat.java Modified: hadoop/core/branches/branch-0.18/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/CHANGES.txt?rev=669474&r1=669473&r2=669474&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/CHANGES.txt (original) +++ hadoop/core/branches/branch-0.18/CHANGES.txt Thu Jun 19 05:51:59 2008 @@ -625,6 +625,9 @@ HADOOP-3533. Add deprecated methods to provide API compatibility between 0.18 and 0.17. Remove the deprecated methods in trunk. (omalley) + HADOOP-3580. Fixes a problem to do with specifying a har as an input to + a job. (Mahadev Konar via ddas) + Release 0.17.1 - Unreleased INCOMPATIBLE CHANGES Modified: hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/HarFileSystem.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/HarFileSystem.java?rev=669474&r1=669473&r2=669474&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/HarFileSystem.java (original) +++ hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/HarFileSystem.java Thu Jun 19 05:51:59 2008 @@ -343,8 +343,24 @@ } FileStatus fsFile = fs.getFileStatus(new Path(archivePath, harStatus.getPartName())); - return fs.getFileBlockLocations(fsFile, - harStatus.getStartIndex(), harStatus.getLength()); + BlockLocation[] rawBlocks = fs.getFileBlockLocations(fsFile, + harStatus.getStartIndex() + start, len); + return fakeBlockLocations(rawBlocks, harStatus.getStartIndex()); + } + + /** + * fake the rawblocks since map reduce uses the block offsets to + * fo some computations regarding the blocks + * @param rawBlocks the raw blocks returned by the filesystem + * @return faked blocks with changed offsets. + */ + private BlockLocation[] fakeBlockLocations(BlockLocation[] rawBlocks, + long startIndex) { + for (BlockLocation block : rawBlocks) { + long rawOffset = block.getOffset(); + block.setOffset(rawOffset - startIndex); + } + return rawBlocks; } /** Modified: hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/FileInputFormat.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/FileInputFormat.java?rev=669474&r1=669473&r2=669474&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/FileInputFormat.java (original) +++ hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/FileInputFormat.java Thu Jun 19 05:51:59 2008 @@ -331,7 +331,7 @@ return i; } } - BlockLocation last = blkLocations[blkLocations.length]; + BlockLocation last = blkLocations[blkLocations.length -1]; long fileLength = last.getOffset() + last.getLength() -1; throw new IllegalArgumentException("Offset " + offset + " is outside of file (0.." +