Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
From: ankurcha <git@git.apache.org>
To: issues@flink.incubator.apache.org
Reply-To: issues@flink.incubator.apache.org
Message-ID: <git-pr-948-flink@git.apache.org>
Subject: [GitHub] flink pull request: [FLINK-1984] Integrate Flink with Apache
 Mesos
Content-Type: text/plain
Date: Tue, 28 Jul 2015 23:11:18 +0000 (UTC)

GitHub user ankurcha opened a pull request:

    https://github.com/apache/flink/pull/948

    [FLINK-1984] Integrate Flink with Apache Mesos

    This pull requests adds a mesos scheduler and an executor (inspired from the work done in, now abandoned, PR #251). The highlights are as follows:
    
    * The mesos scheduler is starts a jobManager in a parallel thread based on the configuration provided using `--configDir` argument.
    * The mesos scheduler is not HA and should be used with marathon or similar service to ensure that there is always one instance running. This may be addressed in future patches.
    * The mesos scheduler uses the some new properties which can be set using the `conf/flink-conf.yaml`. The configuration directory can be specified using the `--confDir` command line argument and a list of configuration values is present in `flink-mesos/src/main/scala/org/apache/flink/mesos/scheduler/package.scala`.
    
    Example usage:
    
    ```bash
    mvn clean package -DskipTests -Pinclude-mesos
    java -Dlog4j.configuration=file:/vagrant/log4j.properties -cp *.jar org.apache.flink.mesos.scheduler.FlinkScheduler --conf-dir /vagrant
    ```
    
    ```yaml
    # flink-conf.yaml
    
    flink.mesos.master: zk://127.0.0.1:2181/mesos
    flink.uberjar.location: file:///vagrant/flink-dist-0.10-SNAPSHOT.jar
    flink.mesos.taskmanagers.mem: 512
    flink.mesos.taskmanagers.cpu: 0.5
    taskmanager.logging.level: INFO
    streamingMode: streaming
    
    jobmanager.web.port: 8081
    webclient.port: 8080
    ```
    
    ```
    # log4j.properties
    
    log4j.rootLogger=INFO, console
    
    log4j.appender.console=org.apache.log4j.ConsoleAppender
    log4j.appender.console.layout=org.apache.log4j.PatternLayout
    log4j.appender.console.layout.ConversionPattern=%-5p %d{ISO8601} [%t] %c{1}: %m%n
    
    # suppress the warning that hadoop native libraries are not loaded (irrelevant for the client)
    log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF
    log4j.logger.org.apache.flink.mesos=DEBUG
    
    # suppress the irrelevant (wrong) warnings from the netty channel handler
    log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, console
    
    ```
    
    An easy way to test this is using https://github.com/mesosphere/playa-mesos to start a mesos cluster in virtualbox (using vagrant) and copying the uberjar to /vagrant and running the above command. The job manager web UI url is set as the mesos framework url.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ankurcha/flink flink-mesos

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/948.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #948
    
----
commit 8a9e3791a2c3946f1adefe64845f16f89e092c77
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-20T08:45:41Z

    wip

commit a937ff65d72a938814162de1e9d97640c566b3f1
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-25T19:18:42Z

    WIP - flink mesos integration initial commit

commit 3f0237adb7b1f45a1fd80e025aa5b511f92a209c
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-25T19:33:57Z

    Add todo for config changes/issues

commit d9c29e2a3174cb90626c9deccc1604e0e010d965
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-27T00:03:12Z

    Fix some stuff

commit 9910f50c11c81631ff8bb351cdf394cda753ff98
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T01:41:20Z

    Add flink-mesos to dist, update scheduler and executor code

commit 4f2d8fed5b2bf8f49eb4da7cd148e83016206330
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T03:14:16Z

    Clean up code + codestyle changes

commit b1613a965a59f89aa2e9c96dcec52fc40847423a
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T07:12:55Z

    Clean up executor, adjust resource defaults

commit 5d39b1524b9ec565c5be28abcd4f80fab928042b
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T20:56:24Z

    Remove status pinging and fix logging level in TaskManagerExecutor

commit 3331596c957ad774ffca31bb2c959b22422560d2
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T22:33:54Z

    Code cleanup and refactoring of the conf classes

commit 33f3597f1c2aa0c7582025d9c242d4aafd9cfa63
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T22:36:04Z

    Remove unused dependencies and plugins

commit 36846a95a64b9700d37954a2d0176b222b6adbe0
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T22:40:50Z

    Use tab based indents in pom files

commit 52904c8593377c349bccf599b863f2a40df0f562
Author: Ankur Chauhan <achauhan@brightcove.com>
Date:   2015-07-28T23:06:30Z

    Add comments for each of the properties

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---