Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Fri, 19 Feb 2016 12:48:18 +0000 (UTC)
From: "Till Rohrmann (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12940147.1455801028000.93072.1455886098128@Atlassian.JIRA>
In-Reply-To: <JIRA.12940147.1455801028000@Atlassian.JIRA>
References: <JIRA.12940147.1455801028000@Atlassian.JIRA>
 <JIRA.12940147.1455801028107@arcas>
Subject: [jira] [Commented] (FLINK-3441) Revisit quickstarts exclusion
 policy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FLINK-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154177#comment-15154177 ] 

Till Rohrmann commented on FLINK-3441:
--------------------------------------

Yes the "provided" approach would be the cleanest in my opinion. With SBT you effectively do it as you've described it with IntelliJ. You have a special module which sets automatically all provided dependencies to "compile". So in order to run your job locally you have to choose the classpath of that module.

Concerning possible problems with incompatible classes, I think that shading away all of Hadoop's dependencies would make it sound.

> Revisit quickstarts exclusion policy
> ------------------------------------
>
>                 Key: FLINK-3441
>                 URL: https://issues.apache.org/jira/browse/FLINK-3441
>             Project: Flink
>          Issue Type: Improvement
>          Components: Quickstarts
>    Affects Versions: 1.0.0
>            Reporter: Till Rohrmann
>            Priority: Minor
>
> Flink provides quickstart projects to quickly set up a Flink project. In order to decrease the size of the resulting jar, the predefined {{pom.xml}} file contains a list of excluded dependencies.
> The problem with this approach is two-fold: First of all, the list seems to be rather arbitrary. Why excluding {{org.apache.commons:commons-math}} but not {{org.apache.commons:commons-math3}}? It seems as if this list would have to be kept up to date with every changing dependency of Flink.
> Secondly and the more severe problem is the following: The exclusions assume that Flink always provides a compatible version for an excluded dependency. However, since the exclusions are not bound to a specific version, this won't necessarily be true. For example, {{snappy-java}} is excluded from the user job jar, because Flink with Hadoop {{2.3.0}} comes with {{snappy-java:1.0.5}}. However, the {{kafka-clients:0.9.0}} which comes with the {{flink-connector-kafka-0.9}} dependency has a transitive dependency on {{snappy-java:1.1.1.7}}. Consequently, this dependency will be excluded from the user job jar per default. If version {{1.1.1.7}} is not compatible to {{snappy-java:1.0.5}}, then the program will crash at runtime. This will be very hard to understand for the user because it is a very subtle bug since the exclusions also affect the transitive dependencies.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)