Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 11BD310151 for ; Sat, 15 Feb 2014 09:02:27 +0000 (UTC) Received: (qmail 9044 invoked by uid 500); 15 Feb 2014 09:02:22 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 8422 invoked by uid 500); 15 Feb 2014 09:02:21 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 8394 invoked by uid 500); 15 Feb 2014 09:02:20 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 8383 invoked by uid 99); 15 Feb 2014 09:02:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Feb 2014 09:02:20 +0000 Date: Sat, 15 Feb 2014 09:02:20 +0000 (UTC) From: "Prasanth J (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-5922) In orc.InStream.CompressedStream, the desired position passed to seek can equal offsets[i] + bytes[i].remaining() when ORC predicate pushdown is enabled MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902355#comment-13902355 ] Prasanth J commented on HIVE-5922: ---------------------------------- Hi Puneeth The issue might be related to https://issues.apache.org/jira/browse/HIVE-6320 or https://issues.apache.org/jira/browse/HIVE-6287 depends on whether you have enable vectorization or not. Is this issue happening in trunk? > In orc.InStream.CompressedStream, the desired position passed to seek can equal offsets[i] + bytes[i].remaining() when ORC predicate pushdown is enabled > -------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: HIVE-5922 > URL: https://issues.apache.org/jira/browse/HIVE-5922 > Project: Hive > Issue Type: Bug > Components: File Formats > Reporter: Yin Huai > > Two stack traces ... > {code} > java.io.IOException: IO error in map input file hdfs://10.38.55.204:8020/user/hive/warehouse/ssdb_bin_compress_orc_large_0_13.db/cycle/000004_0 > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236) > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:210) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.io.IOException: java.io.IOException: Seek outside of data in compressed stream Stream for column 9 kind DATA position: 21496054 length: 33790900 range: 2 offset: 1048588 limit: 1048588 range 0 = 13893791 to 1048588; range 1 = 17039555 to 1310735; range 2 = 20447466 to 1048588; range 3 = 23855377 to 1048588; range 4 = 27263288 to 1048588; range 5 = 30409052 to 1310735 uncompressed: 262144 to 262144 to 21496054 > at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) > at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) > at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:276) > at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) > at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) > at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108) > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:230) > ... 9 more > Caused by: java.io.IOException: Seek outside of data in compressed stream Stream for column 9 kind DATA position: 21496054 length: 33790900 range: 2 offset: 1048588 limit: 1048588 range 0 = 13893791 to 1048588; range 1 = 17039555 to 1310735; range 2 = 20447466 to 1048588; range 3 = 23855377 to 1048588; range 4 = 27263288 to 1048588; range 5 = 30409052 to 1310735 uncompressed: 262144 to 262144 to 21496054 > at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.seek(InStream.java:328) > at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:161) > at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:205) > at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readInts(SerializationUtils.java:450) > at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readDirectValues(RunLengthIntegerReaderV2.java:240) > at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:53) > at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:288) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:510) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1581) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2707) > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:110) > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:86) > at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274) > ... 13 more > {\code} > {code} > java.io.IOException: IO error in map input file hdfs://10.38.55.204:8020/user/hive/warehouse/ssdb_bin_compress_orc_large_0_13.db/cycle/000095_0 > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236) > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:210) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.io.IOException: java.lang.IllegalStateException: Can't read header at compressed stream Stream for column 9 kind DATA position: 20447466 length: 20958101 range: 6 offset: 1835029 limit: 1835029 range 0 = 0 to 524294; range 1 = 1835029 to 2097176; range 2 = 5242940 to 1835029; range 3 = 8650851 to 1835029; range 4 = 11796615 to 2097176; range 5 = 15204526 to 2097176; range 6 = 18612437 to 1835029 uncompressed: 262144 to 262144 > at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) > at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) > at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:276) > at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) > at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) > at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108) > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:230) > ... 9 more > Caused by: java.lang.IllegalStateException: Can't read header at compressed stream Stream for column 9 kind DATA position: 20447466 length: 20958101 range: 6 offset: 1835029 limit: 1835029 range 0 = 0 to 524294; range 1 = 1835029 to 2097176; range 2 = 5242940 to 1835029; range 3 = 8650851 to 1835029; range 4 = 11796615 to 2097176; range 5 = 15204526 to 2097176; range 6 = 18612437 to 1835029 uncompressed: 262144 to 262144 > at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:195) > at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:205) > at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readInts(SerializationUtils.java:450) > at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readDirectValues(RunLengthIntegerReaderV2.java:240) > at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:53) > at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:288) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:510) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1581) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2707) > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:110) > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:86) > at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274) > ... 13 more > {\code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)