hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
Date Mon, 17 Oct 2016 20:31:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583356#comment-15583356
] 

Xuefu Zhang commented on HIVE-14029:
------------------------------------

[~spena], Keeping b/c is a good thing in general. Before we take the effort (which seems a
lot) to do it, I think we should clearly understand and define what b/c is in this case. Spark
is rapidly releasing w/o much b/c in mind. So far, Hive on Spark has once depended on Spark
1.2, 1.3, 1.4, 1.5, and 1.6. I'm not sure what versions of Spark Hive has been released with,
but one thing is clear, Spark isn't b/c between these releases. Before Spark community has
a good sense of keeping b/c in their APIs, it's going to be very hard and burdensome for Hive
to maintain support for different Spark releases, not to mention the library dependency issues
we have had.

I'm okay to start thinking of a shim layer to support multiple versions of Spark, but it sounds
daunting to me due to the dynamics of Spark project.

> Update Spark version to 2.0.0
> -----------------------------
>
>                 Key: HIVE-14029
>                 URL: https://issues.apache.org/jira/browse/HIVE-14029
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ferdinand Xu
>            Assignee: Ferdinand Xu
>              Labels: Incompatible, TODOC2.2
>             Fix For: 2.2.0
>
>         Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, HIVE-14029.3.patch, HIVE-14029.4.patch,
HIVE-14029.5.patch, HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0
to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long
type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message