Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A4C12200B3B for ; Mon, 11 Jul 2016 14:05:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A352D160A85; Mon, 11 Jul 2016 12:05:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id ECD6D160A62 for ; Mon, 11 Jul 2016 14:05:11 +0200 (CEST) Received: (qmail 18809 invoked by uid 500); 11 Jul 2016 12:05:11 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 18789 invoked by uid 99); 11 Jul 2016 12:05:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jul 2016 12:05:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id F1AD72C02A3 for ; Mon, 11 Jul 2016 12:05:10 +0000 (UTC) Date: Mon, 11 Jul 2016 12:05:10 +0000 (UTC) From: "Yibing Shi (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-14205) Hive doesn't support union type with AVRO file format MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 11 Jul 2016 12:05:12 -0000 [ https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370611#comment-15370611 ] Yibing Shi commented on HIVE-14205: ----------------------------------- Will submit a patch later > Hive doesn't support union type with AVRO file format > ----------------------------------------------------- > > Key: HIVE-14205 > URL: https://issues.apache.org/jira/browse/HIVE-14205 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers > Reporter: Yibing Shi > > Reproduce steps: > {noformat} > hive> CREATE TABLE avro_union_test > > PARTITIONED BY (p int) > > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > > STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > > OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > > TBLPROPERTIES ('avro.schema.literal'='{ > > "type":"record", > > "name":"nullUnionTest", > > "fields":[ > > { > > "name":"value", > > "type":[ > > "null", > > "int", > > "long" > > ], > > "default":null > > } > > ] > > }'); > OK > Time taken: 0.105 seconds > hive> alter table avro_union_test add partition (p=1); > OK > Time taken: 0.093 seconds > hive> select * from avro_union_test; > FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet.java.lang.RuntimeException: Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet. > at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140) > at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149) > at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187) > at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220) > at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200) > at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219) > at org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581) > at org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172) > at org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140) > at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > {noformat} > Another test case to show this problem is: > {noformat} > hive> create table avro_union_test2 (value uniontype) stored as avro; > OK > Time taken: 0.053 seconds > hive> show create table avro_union_test2; > OK > CREATE TABLE `avro_union_test2`( > `value` uniontype COMMENT '') > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > 'hdfs://localhost/user/hive/warehouse/avro_union_test2' > TBLPROPERTIES ( > 'transient_lastDdlTime'='1468173589') > Time taken: 0.051 seconds, Fetched: 12 row(s) > {noformat} > Although column {{value}} is defined as {{uniontype}} in create table command, its type becomes {{uniontype}} after table is defined. Hive accidentally make the nullable definition in avro schema ({{\["null", "int", "long"\]}}) into union definition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)