spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-4048) Enhance and extend hadoop-provided profile
Date Mon, 01 Jun 2015 21:03:18 GMT


Marcelo Vanzin commented on SPARK-4048:

bq. because Spark Master is explicitly using code from curator jar.

But that's not a valid argument. spark-core explicitly uses a bunch of Hadoop APIs, and still
if you enable hadoop-provided, those APIs will not be in the Spark assembly.

Again, that is *the whole purpose* of this profile.

> Enhance and extend hadoop-provided profile
> ------------------------------------------
>                 Key: SPARK-4048
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build
>    Affects Versions: 1.2.0
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>             Fix For: 1.3.0
> The hadoop-provided profile is used to not package Hadoop dependencies inside the Spark
assembly. It works, sort of, but it could use some enhancements. A quick list:
> - It doesn't include all things that could be removed from the assembly
> - It doesn't work well when you're publishing artifacts based on it (SPARK-3812 fixes
> - There are other dependencies that could use similar treatment: Hive, HBase (for the
examples), Flume, Parquet, maybe others I'm missing at the moment.
> - Unit tests, more specifically, those that use local-cluster mode, do not work when
the assembly is built with this profile enabled.
> - The scripts to launch Spark jobs do not add needed "provided" jars to the classpath
when this profile is enabled, leaving it for people to figure that out for themselves.
> - The examples assembly duplicates a lot of things in the main assembly.
> Part of this task is selfish since we build internally with this profile and we'd like
to make it easier for us to merge changes without having to keep too many patches on top of
upstream. But those feel like good improvements to me, regardless.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message