hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10161) LLAP: ORC file contains compression buffers larger than bufferSize (OR reader has a bug)
Date Thu, 27 Jun 2019 07:23:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873879#comment-16873879
] 

Wei Zhang commented on HIVE-10161:
----------------------------------

[~harishk15]

Having the similar scenario with you. We backuped a portion of partitions of the partitioned
table to another cluster and encrypted the data. Original table stays managed. When reading
from source cluster backuped partition, the same error happens.

> LLAP: ORC file contains compression buffers larger than bufferSize (OR reader has a bug)
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-10161
>                 URL: https://issues.apache.org/jira/browse/HIVE-10161
>             Project: Hive
>          Issue Type: Sub-task
>    Affects Versions: llap
>            Reporter: Gopal V
>            Assignee: Sergey Shelukhin
>            Priority: Major
>             Fix For: llap
>
>
> The EncodedReaderImpl will die when reading from the cache, when reading data written
by the regular ORC writer 
> {code}
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: Buffer size too small.
size = 262144 needed = 3919246
>         at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:249)
>         at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:201)
>         at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:140)
>         at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:96)
>         at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
>         ... 22 more
> Caused by: java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed
= 3919246
>         at org.apache.hadoop.hive.ql.io.orc.InStream.addOneCompressionBuffer(InStream.java:780)
>         at org.apache.hadoop.hive.ql.io.orc.InStream.uncompressStream(InStream.java:628)
>         at org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:309)
>         at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:278)
>         at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:48)
>         at org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
>         ... 4 more
> ]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex vertex_1424502260528_1945_1_00
[Map 1] killed/failed due to:null]
> {code}
> Turning off hive.llap.io.enabled makes the error go away.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message