hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
Date Sat, 24 Sep 2016 14:41:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15519134#comment-15519134
] 

Rui Li commented on HIVE-14029:
-------------------------------

[~Ferd], you mean unit test will fail right? I don't see failed spark qtest in last QA report.
For qtest/runtime, we use spark-submit to submit the app, so spark should add all its dependencies
to classpath. One problem I can think of is if we don't exclude the glassfish jersey, will
hive pull two versions of jersey into its classpath, i.e. the lib dir? If so, that can cause
problem for hive's functionalities that depend on jersey.

> Update Spark version to 2.0.0
> -----------------------------
>
>                 Key: HIVE-14029
>                 URL: https://issues.apache.org/jira/browse/HIVE-14029
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ferdinand Xu
>            Assignee: Ferdinand Xu
>         Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, HIVE-14029.3.patch, HIVE-14029.4.patch,
HIVE-14029.5.patch, HIVE-14029.6.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0
to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long
type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message