spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [2/2] git commit: Merge pull request #23 from jerryshao/multi-user
Date Thu, 07 Nov 2013 07:22:56 GMT
Merge pull request #23 from jerryshao/multi-user

Add Spark multi-user support for standalone mode and Mesos

This PR add multi-user support for Spark both standalone mode and Mesos (coarse and fine grained
) mode, user can specify the user name who submit app through environment variable `SPARK_USER`
or use default one. Executor will communicate with Hadoop using  specified user name.

Also I fixed one bug in JobLogger when different user wrote job log to specified folder which
has no right file  permission.

I separate previous [PR750]( into two PRs, in this
PR I only solve multi-user support problem. I will try to solve security auth problem in subsequent
PR because security auth is a complicated problem especially for Shark Server like long-run
app (both Kerberos TGT and HDFS delegation token should be renewed or re-created through app's
run time).


Branch: refs/heads/master
Commit: be7e8da98ad04d66b61cd7fc8af7ae61a649d71c
Parents: aadeda5 12dc385
Author: Reynold Xin <>
Authored: Wed Nov 6 23:22:47 2013 -0800
Committer: Reynold Xin <>
Committed: Wed Nov 6 23:22:47 2013 -0800

 .../scala/org/apache/spark/SparkContext.scala   |  10 +
 .../apache/spark/deploy/SparkHadoopUtil.scala   |  18 +-
 .../org/apache/spark/executor/Executor.scala    |   7 +-
 .../org/apache/spark/scheduler/JobLogger.scala  | 757 ++++++++++---------
 .../apache/spark/scheduler/JobLoggerSuite.scala |   4 +-
 5 files changed, 417 insertions(+), 379 deletions(-)

View raw message