hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitoshi Ozawa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12175) Upgrade Kryo version to 3.0.x
Date Wed, 11 Nov 2015 20:48:11 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001057#comment-15001057
] 

Hitoshi Ozawa commented on HIVE-12175:
--------------------------------------

Would really like to this issue get resolved. Until Kryo is upgraded to 3.x, it's really not
usable - not only in Hive but in other OSS projects as well.
Chill is willing to upgrade if Hive and Spark will upgrade to the same version. I've added
a link to Spark issue to upgrade their Kryo to 3.x. There's currently a discussion in Spark
dev ml about Spark 2.0 and proposing upgrading to Kryo 3.x in that release.

https://github.com/twitter/chill/pull/230#issuecomment-155845959

> Upgrade Kryo version to 3.0.x
> -----------------------------
>
>                 Key: HIVE-12175
>                 URL: https://issues.apache.org/jira/browse/HIVE-12175
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>    Affects Versions: 2.0.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-12175.1.patch
>
>
> Current version of kryo (2.22) has some issue (refer exception below and in HIVE-12174)
with serializing ArrayLists generated using Arrays.asList(). We need to either replace all
occurrences of  Arrays.asList() or change the current StdInstantiatorStrategy. This issue
is fixed in later versions and kryo community recommends using DefaultInstantiatorStrategy
with fallback to StdInstantiatorStrategy. More discussion about this issue is here https://github.com/EsotericSoftware/kryo/issues/216.
Alternatively, custom serilization/deserilization class can be provided for Arrays.asList.
> Also, kryo 3.0 introduced unsafe based serialization which claims to have much better
performance for certain types of serialization. 
> Exception:
> {code}
> Caused by: java.lang.NullPointerException
> 	at java.util.Arrays$ArrayList.size(Arrays.java:2847)
> 	at java.util.AbstractList.add(AbstractList.java:108)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	... 57 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message