hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Sankar Sivarama Subramaniyan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-5199) Custom SerDe containing a nonSettable complex data type row object inspector throws cast exception with HIVE 0.11
Date Thu, 05 Sep 2013 02:02:54 GMT

     [ https://issues.apache.org/jira/browse/HIVE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hari Sankar Sivarama Subramaniyan updated HIVE-5199:
----------------------------------------------------

    Description: 
The issue happens because of the changes in HIVE-3833.

Consider a partitioned table with different custom serdes for the partition and tables. The
serde at table level, say, customSerDe1's object inspector is of settableDataType where as
the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType.
The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types
which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI().
However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector
inside  ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

The below exception can happen via FetchOperator as well as MapOperator. 
For example, consider the FetchOperator.
Inside FetchOperator consider the following call:
getRecordReader()->ObjectInspectorConverters. getConverter()

The stack trace as follows:
2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with
exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)

  was:
The issue happens because of the changes in HIVE-3833.

Consider a partitioned table with different custom serdes for the partition and tables. The
serde at table level, say, customSerDe1's object inspector is of settableDataType where as
the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType.
The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types
which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI().
However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector
inside  ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

The below exception can happen via FetchOperator as well as MapOperator. 
For example, consider the FetchOperator.
Inside FetchOperator consider the following call:
getRecordReader()->ObjectInspectorConverters. getConverter()
The output object inspector is of settable type(because it is generated via ObjectInspectorConverters.getConvertedOI())
where as the input object inspector that gets passed  as serde.getObjectorInspector() and
is non-settable. Inside  getConverter(), the (inputOI.equals(outputOI)) check fails and the
switch statement tries to cast the non-settable object inspector to a settable object inspector.
 

The stack trace as follows:
2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with
exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)

    
> Custom SerDe containing a nonSettable complex data type row object inspector throws cast
exception with HIVE 0.11
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-5199
>                 URL: https://issues.apache.org/jira/browse/HIVE-5199
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.11.0
>            Reporter: Hari Sankar Sivarama Subramaniyan
>            Assignee: Hari Sankar Sivarama Subramaniyan
>            Priority: Critical
>         Attachments: HIVE-5199.patch.txt
>
>
> The issue happens because of the changes in HIVE-3833.
> Consider a partitioned table with different custom serdes for the partition and tables.
The serde at table level, say, customSerDe1's object inspector is of settableDataType where
as the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType.
The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types
which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI().
However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector
inside  ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).
> The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.
> The below exception can happen via FetchOperator as well as MapOperator. 
> For example, consider the FetchOperator.
> Inside FetchOperator consider the following call:
> getRecordReader()->ObjectInspectorConverters. getConverter()
> The stack trace as follows:
> 2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed
with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
> java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message