Return-Path: X-Original-To: apmail-spark-reviews-archive@minotaur.apache.org Delivered-To: apmail-spark-reviews-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CC2A217A20 for ; Mon, 20 Apr 2015 20:58:03 +0000 (UTC) Received: (qmail 89697 invoked by uid 500); 20 Apr 2015 20:58:03 -0000 Delivered-To: apmail-spark-reviews-archive@spark.apache.org Received: (qmail 89673 invoked by uid 500); 20 Apr 2015 20:58:03 -0000 Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@spark.apache.org Received: (qmail 89657 invoked by uid 99); 20 Apr 2015 20:58:03 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Apr 2015 20:58:03 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 60604E0921; Mon, 20 Apr 2015 20:58:03 +0000 (UTC) From: vanzin To: reviews@spark.apache.org Reply-To: reviews@spark.apache.org References: In-Reply-To: Subject: [GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap... Content-Type: text/plain Message-Id: <20150420205803.60604E0921@git1-us-west.apache.org> Date: Mon, 20 Apr 2015 20:58:03 +0000 (UTC) Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28728492 --- Diff: docs/security.md --- @@ -31,6 +31,7 @@ SSL must be configured on each node and configured for each component involved i ### YARN mode The key-store can be prepared on the client side and then distributed and used by the executors as the part of the application. It is possible because the user is able to deploy files before the application is started in YARN by using `spark.yarn.dist.files` or `spark.yarn.dist.archives` configuration settings. The responsibility for encryption of transferring these files is on YARN side and has nothing to do with Spark. +For long-running apps like Spark Streaming apps be able to write to HDFS, it is possible to pass a principal and keytab to `spark-submit` via the `--principal` and `--keytab` parameters respectively. The keytab passed in will be copied over to the machine running the Application Master securely via the Hadoop Distributed Cache. The Kerberos login will be periodically renewed using this principal and keytab and the delegation tokens required for HDFS will be generated periodically so the application can continue writing to HDFS. --- End diff -- "...apps like Spark Streaming to be able..." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastructure@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org For additional commands, e-mail: reviews-help@spark.apache.org