flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7778) Relocate ZooKeeper
Date Tue, 31 Oct 2017 14:29:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226875#comment-16226875

ASF GitHub Bot commented on FLINK-7778:

GitHub user zentol opened a pull request:


    [FLINK-7778] [build] Shade ZooKeeper dependency

    ## What is the purpose of the change
    This PR shades and relocates curator&zookeeper into flink-runtime to avoid dependency
conflicts, primarily with hadoop.
    This PR is based on work done by @StephanEwen. I only resolved the remaining issues:
    * flink-mesos did not compile since it relies on the flink-runtime zookeeper dependency,
which was now relocated (the process of which is not visible to flink-mesos). This was fixed
by adding a relocation to flink-mesos for zookeeper that must at all times be in sync. A cleaner
solution would be to create a dedicated flink-shaded-curator/zookeeper module that both flink-runtime/mesos
work against.
    * flink-tests test execution failed since it couldn't find a relocated curator test class.
The relocation pattern for curator in flink-runtime was also applied to curator-test; however
the relocated version is not included in the test jar and was thus not accessible to other
modules. This was fixed by adding an relocation exclusion for classes from `org.apache.curator.test`.
    * flink-yarn test execution failed since it couldn't find the unshaded guava Function
class. Curator is indeed shading&relocating guava, unfortunately not all of it. As such
it was needed to re-add flink-shaded-curator-recipes including the guava relocation. While
it would have been possible to do the relocation in flink-runtime I instead opted for the
known working solution.
    ## Brief change log
    * shade&relocate zookeeper and flink-shaded-curator recipes into flink-runtime
    * add a relocation for zookeeper to flink-mesos, in sync with flink-runtime
    * remove flink-shaded-curator-test, since it is simply no longer needed
    * add travis checks to ensure no unshaded curator/zk make it into flink-dist
    * JobManagerHAJobGraphRecoveryITCase was moved to flink-runtime to resolve a ClassCastException
related to ZK or smth
    ## Verifying this change
    I've only made sure that things compile and tests can be run successfully. I have not
tried this in an actual cluster setting due to time constraints.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (yes)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes)
    ## Documentation
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zentol/flink new_zkcur

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4927
commit b69c21c36443355be95246b3fda795df4fb6524c
Author: Stephan Ewen <sewen@apache.org>
Date:   2017-10-13T20:14:09Z

    [FLINK-7778] [build] Shade ZooKeeper dependency
    Shading the ZooKeeper dependency makes sure that this specific version of
    ZooKeeper is used by the Flink runtime module. The ZooKeeper version is
    sensitive, because we depend on bug fixes in later ZooKeeper versions
    for Flink's high availability.
    This prevents situations where for example a set of added dependencies (for
    example transtive dependencies of Hadoop) cause a different ZooKeeper version
    to be in the classpath and be loaded.
    This commit also removes the 'flink-shaded-curator' module, which was originally
    created to shade guava within curator, but is now obsolete, because newer
    versions of curator shade guava already.

commit af6dc28cfcd8856ea812d8443c464bb1dedb4315
Author: zentol <chesnay@apache.org>
Date:   2017-10-31T10:26:48Z

    Resolve remaining issues


> Relocate ZooKeeper
> ------------------
>                 Key: FLINK-7778
>                 URL: https://issues.apache.org/jira/browse/FLINK-7778
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Build System
>    Affects Versions: 1.4.0
>            Reporter: Till Rohrmann
>            Priority: Critical
>             Fix For: 1.4.0
> If possible, then we should also try to relocate {{ZooKeeper}} in order to avoid dependency
clashes between Flink's {{ZooKeeper}} and Hadoop's {{ZooKeeper}} dependency.

This message was sent by Atlassian JIRA

View raw message