From common-issues-return-9309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 01:28:22 2010 Return-Path: Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: (qmail 52402 invoked from network); 1 Jul 2010 01:28:22 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 1 Jul 2010 01:28:22 -0000 Received: (qmail 18220 invoked by uid 500); 1 Jul 2010 01:28:22 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 18175 invoked by uid 500); 1 Jul 2010 01:28:21 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 17979 invoked by uid 99); 1 Jul 2010 01:28:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 01:28:21 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 01:28:18 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o611RusO021632 for ; Thu, 1 Jul 2010 01:27:56 GMT Message-ID: <6458645.145611277947676511.JavaMail.jira@thor> Date: Wed, 30 Jun 2010 21:27:56 -0400 (EDT) From: "Greg Roelofs (JIRA)" To: common-issues@hadoop.apache.org Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2 files MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884120#action_12884120 ] Greg Roelofs commented on HADOOP-6835: -------------------------------------- OK, so auto-patch doesn't know about the split projects, and hadoop-mapreduce patches don't work in hadoop-common. I tried moving the test case to hadoop-common/src/test/core/org/apache/hadoop/io/compress, but it depends on too many MR classes to be workable (JobConf, Reporter, RecordReader, FileSplit, InputSplit, FileInputFormat, TextInputFormat). So I guess the procedure is (1) reupload the hadoop-common patch and mark as patch-available [actually, I missed a fix in TestCodec, so I need to upload a new one anyway]; (2) get that past QA and checked in; (3) open a separate MR JIRA, attach the test case, and mark it patch-available; and (4) get _that_ past QA and checked in, too. If there's an easier way, feel free to enlighten me. (On a related note, I was told by Reliable Sources to blame Owen, but he just left town. Coincidence?) > Support concatenated gzip and bzip2 files > ----------------------------------------- > > Key: HADOOP-6835 > URL: https://issues.apache.org/jira/browse/HADOOP-6835 > Project: Hadoop Common > Issue Type: Improvement > Components: io > Affects Versions: 0.20.2 > Reporter: Tom White > Assignee: Greg Roelofs > Fix For: 0.22.0 > > Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch > > > When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.