Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 34E5118ACF for ; Thu, 18 Jun 2015 02:40:01 +0000 (UTC) Received: (qmail 98197 invoked by uid 500); 18 Jun 2015 02:40:01 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 98169 invoked by uid 500); 18 Jun 2015 02:40:01 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 98153 invoked by uid 99); 18 Jun 2015 02:40:01 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2015 02:40:01 +0000 Date: Thu, 18 Jun 2015 02:40:01 +0000 (UTC) From: "Prasanth Jayachandran (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-11031) ORC concatenation of old files can fail while merging column statistics MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-11031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591142#comment-14591142 ] Prasanth Jayachandran commented on HIVE-11031: ---------------------------------------------- [~gopalv] Added some changes to throw when incompatible statistics gets merged. Also the orc files which does not have stripe statistics will be added to incompatible file set (ignored from merging). > ORC concatenation of old files can fail while merging column statistics > ----------------------------------------------------------------------- > > Key: HIVE-11031 > URL: https://issues.apache.org/jira/browse/HIVE-11031 > Project: Hive > Issue Type: Bug > Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 2.0.0 > Reporter: Prasanth Jayachandran > Assignee: Prasanth Jayachandran > Priority: Critical > Attachments: HIVE-11031.2.patch, HIVE-11031.patch > > > Column statistics in ORC are optional protobuf fields. Old ORC files might not have statistics for newly added types like decimal, date, timestamp etc. But column statistics merging assumes column statistics exists for these types and invokes merge. For example, merging of TimestampColumnStatistics directly casts the received ColumnStatistics object without doing instanceof check. If the ORC file contains time stamp column statistics then this will work else it will throw ClassCastException. > Also, the file merge operator swallows the exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)