flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Rohrmann (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3441) Revisit quickstarts exclusion policy
Date Thu, 18 Feb 2016 13:11:18 GMT
Till Rohrmann created FLINK-3441:

             Summary: Revisit quickstarts exclusion policy
                 Key: FLINK-3441
                 URL: https://issues.apache.org/jira/browse/FLINK-3441
             Project: Flink
          Issue Type: Improvement
          Components: Quickstarts
    Affects Versions: 1.0.0
            Reporter: Till Rohrmann
            Priority: Minor

Flink provides quickstart projects to quickly set up a Flink project. In order to decrease
the size of the resulting jar, the predefined {{pom.xml}} file contains a list of excluded

The problem with this approach is two-fold: First of all, the list seems to be rather arbitrary.
Why excluding {{org.apache.commons:commons-math}} but not {{org.apache.commons:commons-math3}}?
It seems as if this list would have to be kept up to date with every changing dependency of

Secondly and the more severe problem is the following: The exclusions assume that Flink always
provides a compatible version for an excluded dependency. However, since the exclusions are
not bound to a specific version, this won't necessarily be true. For example, {{snappy-java}}
is excluded from the user job jar, because Flink with Hadoop {{2.3.0}} comes with {{snappy-java:1.0.5}}.
However, the {{kafka-clients:0.9.0}} which comes with the {{flink-connector-kafka-0.9}} dependency
has a transitive dependency on {{snappy-java:}}. Consequently, this dependency will
be excluded from the user job jar per default. If version {{}} is not compatible to
{{snappy-java:1.0.5}}, then the program will crash at runtime. This will be very hard to understand
for the user because it is a very subtle bug since the exclusions also affect the transitive

This message was sent by Atlassian JIRA

View raw message