Return-Path: Delivered-To: apmail-hadoop-core-dev-archive@www.apache.org Received: (qmail 45858 invoked from network); 16 Mar 2009 23:36:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 16 Mar 2009 23:36:13 -0000 Received: (qmail 25695 invoked by uid 500); 16 Mar 2009 23:36:11 -0000 Delivered-To: apmail-hadoop-core-dev-archive@hadoop.apache.org Received: (qmail 25683 invoked by uid 500); 16 Mar 2009 23:36:11 -0000 Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-dev@hadoop.apache.org Received: (qmail 25672 invoked by uid 99); 16 Mar 2009 23:36:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Mar 2009 16:36:11 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Mar 2009 23:36:10 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 94A81234C04B for ; Mon, 16 Mar 2009 16:35:50 -0700 (PDT) Message-ID: <1506140201.1237246550608.JavaMail.jira@brutus> Date: Mon, 16 Mar 2009 16:35:50 -0700 (PDT) From: "Chris Douglas (JIRA)" To: core-dev@hadoop.apache.org Subject: [jira] Updated: (HADOOP-5281) GzipCodec fails second time it is used in a process In-Reply-To: <1435985498.1235012221897.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5281: ---------------------------------- Description: The attached code (GZt.java) raises: {noformat} java.io.IOException: incorrect header check at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method) at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221) at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319) at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435) at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417) at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404) at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523) {noformat} was: The following code raises: java.io.IOException: incorrect header check at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method) at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221) at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62) at java.io.DataInputStream.readByte(DataInputStream.java:248) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319) at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435) at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417) at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404) at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523) {quote} import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.MapFile; import org.apache.hadoop.io.SequenceFile.CompressionType; import org.apache.hadoop.io.compress.GzipCodec; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; public class GZt extends Configured implements Tool { private FileSystem fileSystem; MapFile.Writer newMap(Path p) throws IOException { return new MapFile.Writer(getConf(), fileSystem, p.toString(), IntWritable.class, IntWritable.class, CompressionType.BLOCK, new GzipCodec(), null); } @Override public int run(String[] args) throws Exception { Path p = new Path(args[0]); fileSystem = FileSystem.get(getConf()); fileSystem.mkdirs(p); for (int i = 0; i < 2; i++) { MapFile.Writer w = newMap(new Path(p, "b" + i)); for (int j = 0; j < 100; j++) { w.append(new IntWritable(j), new IntWritable(j)); } w.close(); } MapFile.Reader rdr = new MapFile.Reader(fileSystem, new Path(p, "b1") .toString(), getConf()); rdr.get(new IntWritable(1), new IntWritable()); return 0; } public static void main(String[] args) throws Exception { int res = ToolRunner.run(new Configuration(), new GZt(), args); System.exit(res); } } {quote} Edited description > GzipCodec fails second time it is used in a process > --------------------------------------------------- > > Key: HADOOP-5281 > URL: https://issues.apache.org/jira/browse/HADOOP-5281 > Project: Hadoop Core > Issue Type: Bug > Affects Versions: 0.19.0 > Environment: 0.19.0 / Linux / amd64 > Reporter: Ben Maurer > Priority: Blocker > Attachments: GZt.java > > > The attached code (GZt.java) raises: > {noformat} > java.io.IOException: incorrect header check > at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method) > at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221) > at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80) > at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74) > at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62) > at java.io.DataInputStream.readByte(DataInputStream.java:248) > at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325) > at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) > at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319) > at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435) > at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417) > at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404) > at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523) > {noformat} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.