hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
Date Tue, 27 Sep 2016 08:00:35 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525416#comment-15525416
] 

Rui Li commented on HIVE-14029:
-------------------------------

I tried the 5th patch locally. The jersey 2 stuff won't be pulled to lib, or the hive-exec
jar. I also tried to identify the minimum set. I managed to run some simple queries (spark
on yarn) with only {{scala-library, spark-core, spark-network-common, spark-network-shuffle}}.
However, if we want to support local mode, we need more jars added to hive's lib, including
the jersey 2. Then we may have conflict problem. Other than that, I think the 5th patch is
enough for us (although I think we should exclude jersey 2 instead of just javax.ws.rs).

If we still want to go the way as the 6th, 7th patches, maybe we can look at how we handle
the guava conflict in the pom of spark-client and do something similar.

> Update Spark version to 2.0.0
> -----------------------------
>
>                 Key: HIVE-14029
>                 URL: https://issues.apache.org/jira/browse/HIVE-14029
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ferdinand Xu
>            Assignee: Ferdinand Xu
>         Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, HIVE-14029.3.patch, HIVE-14029.4.patch,
HIVE-14029.5.patch, HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0
to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long
type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message