spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tgravescs <>
Subject [GitHub] spark pull request #14718: [SPARK-16711] YarnShuffleService doesn't re-init ...
Date Fri, 19 Aug 2016 14:23:58 GMT
GitHub user tgravescs opened a pull request:

    [SPARK-16711] YarnShuffleService doesn't re-init properly on YARN rolling upgrade

    The Spark Yarn Shuffle Service doesn't re-initialize the application credentials early
enough which causes any other spark executors trying to fetch from that node during a rolling
upgrade to fail with "java.lang.NullPointerException: Password cannot be null if SASL is enabled".
 Right now the spark shuffle service relies on the Yarn nodemanager to re-register the applications,
unfortunately this is after we open the port for other executors to connect. If other executors
connected before the re-register they get a null pointer exception which isn't a re-tryable
exception and cause them to fail pretty quickly. To solve this I added another leveldb file
so that it can save and re-initialize all the applications before opening the port for other
executors to connect to it.  Adding another leveldb was simpler from the code structure point
of view.
    Most of the code changes are moving things to common util class.
    Patch was tested manually on a Yarn cluster with rolling upgrade was happing while spark
job was running. Without the patch I consistently get the NullPointerException, with the patch
the job gets a few Connection refused exceptions but the retries kick in and the it succeeds.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-16711

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14718
commit f0a5c564f0a1158f9432524deecd3f64243a673c
Author: Thomas Graves <>
Date:   2016-08-18T18:59:29Z

    [SPARK-16711] YarnShuffleService doesn't re-init properly on YARN rolling upgrade


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message