hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yibing Shi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14205) Hive doesn't support union type with AVRO file format
Date Tue, 12 Jul 2016 11:08:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372699#comment-15372699
] 

Yibing Shi commented on HIVE-14205:
-----------------------------------

code review:
https://reviews.apache.org/r/49952/

> Hive doesn't support union type with AVRO file format
> -----------------------------------------------------
>
>                 Key: HIVE-14205
>                 URL: https://issues.apache.org/jira/browse/HIVE-14205
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Yibing Shi
>            Assignee: Yibing Shi
>         Attachments: HIVE-14205.1.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
>     > PARTITIONED BY (p int)
>     > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>     > STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>     > OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>     > TBLPROPERTIES ('avro.schema.literal'='{
>     >    "type":"record",
>     >    "name":"nullUnionTest",
>     >    "fields":[
>     >       {
>     >          "name":"value",
>     >          "type":[
>     >             "null",
>     >             "int",
>     >             "long"
>     >          ],
>     >          "default":null
>     >       }
>     >    ]
>     > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Failed with
exception Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported
yet.java.lang.RuntimeException: Hive internal error inside isAssignableFromSettablePrimitiveOI
void not supported yet.
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> 	at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:140)
> 	at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> 	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype<int,bigint>) stored as
avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
>   `value` uniontype<void,int,bigint> COMMENT '')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION
>   'hdfs://localhost/user/hive/warehouse/avro_union_test2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1468173589')
> Time taken: 0.051 seconds, Fetched: 12 row(s)
> {noformat}
> Although column {{value}} is defined as {{uniontype<int,bigint>}} in create table
command, its type becomes {{uniontype<void,int,bigint>}} after table is defined. Hive
accidentally make the nullable definition in avro schema ({{\["null", "int", "long"\]}}) 
into union definition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message