spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewor14 <>
Subject [GitHub] spark pull request: [WIP][SPARK-5388] Provide a stable application...
Date Tue, 27 Jan 2015 02:26:18 GMT
GitHub user andrewor14 opened a pull request:

    [WIP][SPARK-5388] Provide a stable application submission gateway

    The goal is to provide a stable application submission gateway that is not inherently
based on Akka, which is unstable across versions. This PR targets standalone cluster mode,
but is implemented in a general enough manner to be extended to other modes in the future.
Client mode is currently not included in the changes here because there are many more Akka
messages exchanged there.
    As of the changes here, the Master will advertise two ports, 7077 and 17077. We need to
keep around the old one (7077) for client mode and older versions of Spark submit. However,
all new versions of Spark submit will use the REST gateway (17077).
    This is still WIP, but comments and feedback are most welcome.

You can merge this pull request into a Git repository by running:

    $ git pull rest

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4216
commit 53e7c0e9a208e70c374793baf693d64fe4e6b8e1
Author: Andrew Or <>
Date:   2015-01-15T00:29:52Z

    Initial client, server, and all the messages
    This commit introduces type-safe schemas for all messages exchanged
    in the REST protocol. Each message is expected to contain an ACTION
    field that has only one possible value for each message type.
    Before the message is sent, we validate that all required fields
    are in fact present, and that the value of the action field is the
    correct type.
    The next step is to actually integrate this in standalone mode.

commit af9d9cb2933774e33278022a38196b522193a766
Author: Andrew Or <>
Date:   2015-01-17T03:12:32Z

    Integrate REST protocol in standalone mode
    This commit embeds the REST server in the standalone Master and
    forces Spark submit to submit applications through the REST client.
    This is the first working end-to-end implementation of a stable
    submission interface in standalone cluster mode.

commit 6ff088dca37f3f888a94c00a5d533ef8c6a6e6f5
Author: Andrew Or <>
Date:   2015-01-20T00:24:05Z

    Rename classes to generalize REST protocol
    Previously the REST protocol was very explicitly tied to the
    standalone mode. This commit frees the protocol from this

commit 484bd2172b847433c989d7c450fbbc99dddb1f56
Author: Andrew Or <>
Date:   2015-01-20T01:02:25Z

    Specify an ordering for fields in SubmitDriverRequestMessage
    will appear in the JSON at random places. Now they are grouped
    together at the end of the JSON blob.

commit e958caec3e2f5bbd1dd2cf6dc0e02a3ad27c2699
Author: Andrew Or <>
Date:   2015-01-20T23:29:56Z

    Supported nested values in messages
    This is applicable to application arguments, Spark properties, and
    environment variables, all of which were previously handled through
    parameterized fields, which were cumbersome to parse. Since JSON
    naturally supports nesting, we should take advantage of it too.
    This commit refactors the code that converts the messages to and
    from JSON in a way that subclasses can easily override the conversion
    behavior without duplicating code.

commit 544de1dd5d4d2bb35180775ddf655bedee4d44ae
Author: Andrew Or <>
Date:   2015-01-21T22:02:33Z

    Major clean ups in code and comments
    This involves refactoring SparkSubmit a little to put the code
    that launches the REST client in the right place. This commit also
    adds port retry logic in the REST server, which was previously

commit 120ab9d33484ccc20f45aee6272daeca5ebcc878
Author: Andrew Or <>
Date:   2015-01-21T22:55:35Z

    Support kill and request driver status through SparkSubmit

commit b44e103b78b36fadd887dc4b894027a03069b1f7
Author: Andrew Or <>
Date:   2015-01-22T01:19:33Z

    Implement status requests + fix validation behavior
    This commit makes the StandaloneRestServer actually handle status
    requests. The existing polling behavior from o.a.s.deploy.Client
    is also implemented in the StandaloneRestClient and amended.
    Additionally, the validation behavior was confusing before this
    commit. Previously the error message would seem to indicate that
    the user constructed a malformed message even if the message was
    constructed on the server side. This commit ensures that the error
    message is different for these two cases.

commit 51c5ca6d8ef448f7b6181c684fa3ee3794f0d6b8
Author: Andrew Or <>
Date:   2015-01-22T01:43:43Z

    Distinguish client and server side Spark versions
    Otherwise it's a little ambiguous what we mean by SPARK_VERSION.

commit 9e21b7294ed6fd4cc62691f52ccf2665dc18536c
Author: Andrew Or <>
Date:   2015-01-23T19:39:48Z

    Action -> SparkSubmitAction (minor)

commit 63c05b3f403626fe8ff4ce16a00047fb7335890c
Author: Andrew Or <>
Date:   2015-01-23T19:48:50Z

    Remove MASTER as a field (minor)

commit 206cae46093cab19e9f066388c6f848df3ef5391
Author: Andrew Or <>
Date:   2015-01-27T00:59:11Z

    Refactor and add tests for the REST protocol
    This commit does the following major things:
      (1) Refactor SparkSubmit such that SparkSubmitSuite now passes
      (2) Refactor the REST messages such that it's easier to test them
      (3) Add type-safety validation logic for REST fields
      (4) Move REST fields to its own file
      (5) Maintain ordering of fields added to REST messages
      (6) Add an option to disable the REST server, as we do in tests

commit 77774ba94e4c5c66af7d5053970ed30ec22ffef3
Author: Andrew Or <>
Date:   2015-01-27T02:11:46Z

    Minor fixes

commit 6568ca534166bdcad7533385ac958d049ca63a28
Author: Andrew Or <>
Date:   2015-01-27T02:20:13Z

    Merge branch 'master' of into rest


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message