Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DB1C411CF6 for ; Sat, 5 Apr 2014 19:03:19 +0000 (UTC) Received: (qmail 98717 invoked by uid 500); 5 Apr 2014 19:03:18 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 98130 invoked by uid 500); 5 Apr 2014 19:03:16 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 98122 invoked by uid 500); 5 Apr 2014 19:03:15 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 98119 invoked by uid 99); 5 Apr 2014 19:03:15 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Apr 2014 19:03:15 +0000 Date: Sat, 5 Apr 2014 19:03:15 +0000 (UTC) From: "Tongjie Chen (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-6784) parquet-hive should allow column type change MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongjie Chen updated HIVE-6784: ------------------------------- Attachment: HIVE-6784.2.patch.txt add a new qtest for changing type > parquet-hive should allow column type change > -------------------------------------------- > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers > Affects Versions: 0.13.0 > Reporter: Tongjie Chen > Fix For: 0.14.0 > > Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt > > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), it will result in exception thrown from SerDe: "org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will try to perform cast (null value in case of incompatible type). > Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored in footers of parquet files); ParquetHiveSerDe also creates an corresponding ArrayWritableObjectInspector (but using column type info from metastore). Whenever there is column type change, the objector inspector will throw exception, since WritableLongObjectInspector cannot inspect an IntWritable etc... > Conversion has to happen somewhere if we want to allow type change. SerDe's deserialize method seems a natural place for it. > Currently, serialize method calls createStruct (then createPrimitive) for every record, but it creates a new object regardless, which seems expensive. I think that could be optimized a bit by just returning the object passed if already of the right type. deserialize also reuse this method, if there is a type change, there will be new object to be created, which I think is inevitable. -- This message was sent by Atlassian JIRA (v6.2#6252)