flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Rohrmann (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3441) Revisit quickstarts exclusion policy
Date Fri, 19 Feb 2016 12:48:18 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154177#comment-15154177

Till Rohrmann commented on FLINK-3441:

Yes the "provided" approach would be the cleanest in my opinion. With SBT you effectively
do it as you've described it with IntelliJ. You have a special module which sets automatically
all provided dependencies to "compile". So in order to run your job locally you have to choose
the classpath of that module.

Concerning possible problems with incompatible classes, I think that shading away all of Hadoop's
dependencies would make it sound.

> Revisit quickstarts exclusion policy
> ------------------------------------
>                 Key: FLINK-3441
>                 URL: https://issues.apache.org/jira/browse/FLINK-3441
>             Project: Flink
>          Issue Type: Improvement
>          Components: Quickstarts
>    Affects Versions: 1.0.0
>            Reporter: Till Rohrmann
>            Priority: Minor
> Flink provides quickstart projects to quickly set up a Flink project. In order to decrease
the size of the resulting jar, the predefined {{pom.xml}} file contains a list of excluded
> The problem with this approach is two-fold: First of all, the list seems to be rather
arbitrary. Why excluding {{org.apache.commons:commons-math}} but not {{org.apache.commons:commons-math3}}?
It seems as if this list would have to be kept up to date with every changing dependency of
> Secondly and the more severe problem is the following: The exclusions assume that Flink
always provides a compatible version for an excluded dependency. However, since the exclusions
are not bound to a specific version, this won't necessarily be true. For example, {{snappy-java}}
is excluded from the user job jar, because Flink with Hadoop {{2.3.0}} comes with {{snappy-java:1.0.5}}.
However, the {{kafka-clients:0.9.0}} which comes with the {{flink-connector-kafka-0.9}} dependency
has a transitive dependency on {{snappy-java:}}. Consequently, this dependency will
be excluded from the user job jar per default. If version {{}} is not compatible to
{{snappy-java:1.0.5}}, then the program will crash at runtime. This will be very hard to understand
for the user because it is a very subtle bug since the exclusions also affect the transitive

This message was sent by Atlassian JIRA

View raw message