spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@apache.org
Subject [2/2] git commit: Merge pull request #23 from jerryshao/multi-user
Date Thu, 07 Nov 2013 07:22:56 GMT
Merge pull request #23 from jerryshao/multi-user

Add Spark multi-user support for standalone mode and Mesos

This PR add multi-user support for Spark both standalone mode and Mesos (coarse and fine grained
) mode, user can specify the user name who submit app through environment variable `SPARK_USER`
or use default one. Executor will communicate with Hadoop using  specified user name.

Also I fixed one bug in JobLogger when different user wrote job log to specified folder which
has no right file  permission.

I separate previous [PR750](https://github.com/mesos/spark/pull/750) into two PRs, in this
PR I only solve multi-user support problem. I will try to solve security auth problem in subsequent
PR because security auth is a complicated problem especially for Shark Server like long-run
app (both Kerberos TGT and HDFS delegation token should be renewed or re-created through app's
run time).


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/be7e8da9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/be7e8da9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/be7e8da9

Branch: refs/heads/master
Commit: be7e8da98ad04d66b61cd7fc8af7ae61a649d71c
Parents: aadeda5 12dc385
Author: Reynold Xin <rxin@apache.org>
Authored: Wed Nov 6 23:22:47 2013 -0800
Committer: Reynold Xin <rxin@apache.org>
Committed: Wed Nov 6 23:22:47 2013 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/SparkContext.scala   |  10 +
 .../apache/spark/deploy/SparkHadoopUtil.scala   |  18 +-
 .../org/apache/spark/executor/Executor.scala    |   7 +-
 .../org/apache/spark/scheduler/JobLogger.scala  | 757 ++++++++++---------
 .../apache/spark/scheduler/JobLoggerSuite.scala |   4 +-
 5 files changed, 417 insertions(+), 379 deletions(-)
----------------------------------------------------------------------



Mime
View raw message