spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-2420) Change Spark build to minimize library conflicts
Date Mon, 28 Jul 2014 22:57:38 GMT


Marcelo Vanzin commented on SPARK-2420:

Just an update from our offline conversation: even if it's a little more work on the Spark
side, I'm fine with shading, given that it would allow users to use newer versions of Guava
without having to worry (too much) about compatibility.

> Change Spark build to minimize library conflicts
> ------------------------------------------------
>                 Key: SPARK-2420
>                 URL:
>             Project: Spark
>          Issue Type: Wish
>          Components: Build
>    Affects Versions: 1.0.0
>            Reporter: Xuefu Zhang
>         Attachments: spark_1.0.0.patch
> During the prototyping of HIVE-7292, many library conflicts showed up because Spark build
contains versions of libraries that's vastly different from current major Hadoop version.
It would be nice if we can choose versions that's in line with Hadoop or shading them in the
assembly. Here are the wish list:
> 1. Upgrade protobuf version to 2.5.0 from current 2.4.1
> 2. Shading Spark's jetty and servlet dependency in the assembly.
> 3. guava version difference. Spark is using a higher version. I'm not sure what's the
best solution for this.
> The list may grow as HIVE-7292 proceeds.
> For information only, the attached is a patch that we applied on Spark in order to make
Spark work with Hive. It gives an idea of the scope of changes.

This message was sent by Atlassian JIRA

View raw message