hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ferdinand Xu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-14029) Update Spark version to 2.0.0
Date Sat, 24 Sep 2016 03:08:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15518237#comment-15518237
] 

Ferdinand Xu edited comment on HIVE-14029 at 9/24/16 3:07 AM:
--------------------------------------------------------------

Hi [~xuefuz] These two dependencies are not required in build. It's required for the runtime.
If you try to run some HoS job, it will fail to create Spark client since API changes in these
two library. You can see failed queries above for the reference.


was (Author: ferd):
Hi [~xuefuz] If you try to run some HoS job, it will fail to create Spark client since API
changes in these two library. I just test some simple queries on my cluster and find these
two dependencies are required. 

> Update Spark version to 2.0.0
> -----------------------------
>
>                 Key: HIVE-14029
>                 URL: https://issues.apache.org/jira/browse/HIVE-14029
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ferdinand Xu
>            Assignee: Ferdinand Xu
>         Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, HIVE-14029.3.patch, HIVE-14029.4.patch,
HIVE-14029.5.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0
to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long
type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message