spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vanzin <>
Subject [GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...
Date Mon, 20 Apr 2015 20:58:03 GMT
Github user vanzin commented on a diff in the pull request:
    --- Diff: docs/ ---
    @@ -31,6 +31,7 @@ SSL must be configured on each node and configured for each component
involved i
     ### YARN mode
     The key-store can be prepared on the client side and then distributed and used by the
executors as the part of the application. It is possible because the user is able to deploy
files before the application is started in YARN by using `spark.yarn.dist.files` or `spark.yarn.dist.archives`
configuration settings. The responsibility for encryption of transferring these files is on
YARN side and has nothing to do with Spark.
    +For long-running apps like Spark Streaming apps be able to write to HDFS, it is possible
to pass a principal and keytab to `spark-submit` via the `--principal` and `--keytab` parameters
respectively. The keytab passed in will be copied over to the machine running the Application
Master securely via the Hadoop Distributed Cache. The Kerberos login will be periodically
renewed using this principal and keytab and the delegation tokens required for HDFS will be
generated periodically so the application can continue writing to HDFS.
    --- End diff --
    "...apps like Spark Streaming to be able..."

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message