spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From devaraj-kavali <>
Subject [GitHub] spark pull request #19616: [SPARK-22404][YARN][WIP] Provide an option to use...
Date Tue, 31 Oct 2017 00:13:38 GMT
GitHub user devaraj-kavali opened a pull request:

    [SPARK-22404][YARN][WIP] Provide an option to use unmanaged AM in yarn-client mode

    ## What changes were proposed in this pull request?
    Providing a new configuration "spark.yarn.un-managed-am" (defaults to false) to enable
the Unmanaged AM Application in Yarn Client mode which launches the Application Master service
as part of the Client. It utilizes the existing code for communicating between the Application
Master <-> Task Scheduler for the container requests/allocations/launch, and eliminates
    1. Allocating and launching the Application Master container
    2. Remote Node/Process communication between Application Master <-> Task Scheduler
    ## How was this patch tested?
    I verified manually running the applications in yarn-client mode with "spark.yarn.un-managed-am"
enabled, and also ensured that there is no impact to the existing execution flows. I am verifying
some more failure scenarios, will update the PR if anything needs to be fixed. I would like
to hear others feedback/thoughts on this.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-22404

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19616
commit e51f99ef04e4fd797f4c715b1773c1d245a8a0cd
Author: Devaraj K <>
Date:   2017-10-31T00:06:48Z

    [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message