Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7BDE31885C for ; Mon, 7 Mar 2016 20:00:42 +0000 (UTC) Received: (qmail 43529 invoked by uid 500); 7 Mar 2016 20:00:42 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 43488 invoked by uid 500); 7 Mar 2016 20:00:42 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 43474 invoked by uid 99); 7 Mar 2016 20:00:42 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2016 20:00:42 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id CD5AF2C1F5C for ; Mon, 7 Mar 2016 20:00:41 +0000 (UTC) Date: Mon, 7 Mar 2016 20:00:41 +0000 (UTC) From: "Prasanth Jayachandran (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183590#comment-15183590 ] Prasanth Jayachandran commented on HIVE-13083: ---------------------------------------------- Thanks! I will update golden files for TestJsonFileDump on commit. > Writing HiveDecimal to ORC can wrongly suppress present stream > -------------------------------------------------------------- > > Key: HIVE-13083 > URL: https://issues.apache.org/jira/browse/HIVE-13083 > Project: Hive > Issue Type: Bug > Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0 > Reporter: Yi Zhang > Assignee: Prasanth Jayachandran > Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch, HIVE-13083.2.patch, HIVE-13083.3.patch, HIVE-13083.4.patch, HIVE-13083.4.patch > > > HIVE-3976 can cause ORC file to be unreadable. The changes introduced in HIVE-3976 for DecimalTreeWriter can create null values after updating the isPresent stream. https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337 > As result of the above return statement, isPresent stream state can become wrong. The isPresent stream thinks all values are non-null and hence suppressed. But the data stream will be of 0 length. When reading such files we will get the following exception > {code} > Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 > at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176) > at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264) > at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004) > at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039) > ... 24 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)