flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2288) Setup ZooKeeper for distributed coordination
Date Fri, 03 Jul 2015 15:03:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613238#comment-14613238
] 

ASF GitHub Bot commented on FLINK-2288:
---------------------------------------

GitHub user uce opened a pull request:

    https://github.com/apache/flink/pull/886

    [wip] [FLINK-2288] [FLINK-2302] Setup ZooKeeper for distributed coordination

    - FLINK-2288: Setup ZooKeeper for distributed coordination
      * Add FlinkZooKeeperQuorumPeer to wrap ZooKeeper's quorum peers with
        utilities to write required config values (default datadir, myid)
      * Add default conf/zoo.cfg config for ZooKeeper
      * Add startup scripts for ZooKeeper quorum
      * Add conf/masters file for HA masters
      * @rmetzger This PR includes docs ;-)
    
    - FLINK-2302: Allow multiple instances to run on single host
      * Multiple TaskManager and JobManager instances can run on a single
        host.
    
    @tillrohrmann, you can base your changes on this branch. After that we can close this
PR. I've added TODOs in TaskManager and JobManager, where you need to integrate your leader
election/retrieval service.
    
    From the docs:
    
    
    ## Example: Start and stop a local HA-cluster with 2 JobManagers
    
    1. **Configure ZooKeeper quorum** in `conf/flink.yaml`:
       
       <pre>ha.zookeeper.quorum: localhost</pre>
    
    2. **Configure masters** in `conf/masters`:
    
       <pre>
    localhost
    localhost</pre>
    
    3. **Configure ZooKeeper server** in `conf/zoo.cfg` (currently it's only possible to run
a single ZooKeeper server per machine, because there is a single client port per configuration):
    
       <pre>server.0=localhost:2888:3888</pre>
    
    4. **Start ZooKeeper quorum**:
    
       <pre>
    $ bin/start-zookeeper-quorum.sh
    Starting zookeeper daemon on host localhost.</pre>
    
    5. **Start an HA-cluster**:
       
       <pre>
    $ bin/start-cluster-streaming.sh
    Starting HA cluster (streaming mode) with 2 masters and 1 peers in ZooKeeper quorum.
    Starting jobmanager daemon on host localhost.
    Starting jobmanager daemon on host localhost.
    Starting taskmanager daemon on host localhost.</pre>
    
    6. **Stop ZooKeeper quorum and cluster**:
    
       <pre>
    $ bin/stop-cluster.sh
    Stopping taskmanager daemon (pid: 7647) on localhost.
    Stopping jobmanager daemon (pid: 7495) on host localhost.
    Stopping jobmanager daemon (pid: 7349) on host localhost.
    $ bin/stop-zookeeper-quorum.sh
    Stopping zookeeper daemon (pid: 7101) on host localhost.</pre>


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/uce/flink zk-2288

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/886.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #886
    
----
commit 2b6910edc6927c32aa7abdc42870ae1a0813a9ae
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-07-03T09:04:45Z

    [FLINK-2288] [FLINK-2302] Setup ZooKeeper for distributed coordination
    
    - FLINK-2288: Setup ZooKeeper for distributed coordination
      * Add FlinkZooKeeperQuorumPeer to wrap ZooKeeper's quorum peers with
        utilities to write required config values (default datadir, myid)
      * Add default conf/zoo.cfg config for ZooKeeper
      * Add startup scripts for ZooKeeper quorum
      * Add conf/masters file for HA masters
    
    - FLINK-2302: Allow multiple instances to run on single host
      * Multiple TaskManager and JobManager instances can run on a single
        host.

commit 7b4cd2c0ac49c120e1b4f97f4888e1d23ce90937
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-07-03T14:45:15Z

    [FLINK-2288] [docs] Add docs for HA/ZooKeeper setup

----


> Setup ZooKeeper for distributed coordination
> --------------------------------------------
>
>                 Key: FLINK-2288
>                 URL: https://issues.apache.org/jira/browse/FLINK-2288
>             Project: Flink
>          Issue Type: Sub-task
>          Components: JobManager, TaskManager
>            Reporter: Ufuk Celebi
>            Assignee: Ufuk Celebi
>             Fix For: 0.10
>
>
> Having standby JM instances for job manager high availabilty requires distributed coordination
between JM, TM, and clients. For this, we will use ZooKeeper (ZK).
> Pros:
> - Proven solution (other projects use it for this as well)
> - Apache TLP with large community, docs, and library with required "recipies" like leader
election (see below)
> Related Wiki: https://cwiki.apache.org/confluence/display/FLINK/JobManager+High+Availability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message