spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ma...@apache.org
Subject [4/4] git commit: Merge pull request #149 from tgravescs/fixSecureHdfsAccess
Date Sat, 09 Nov 2013 21:48:07 GMT
Merge pull request #149 from tgravescs/fixSecureHdfsAccess

Fix secure hdfs access for spark on yarn

https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it
works with secure hdfs on standalone. Fixing it at least for spark on yarn.

The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account
things calling the getPartitions before sparkContext is initialized. The DAGScheduler does
this as it tries to getShuffleMapStage.


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8af99f23
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8af99f23
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/8af99f23

Branch: refs/heads/master
Commit: 8af99f2356ed19fe43d722ada02f8802cbd46d40
Parents: 72a601e 13a1950
Author: Matei Zaharia <matei@eecs.berkeley.edu>
Authored: Sat Nov 9 13:48:00 2013 -0800
Committer: Matei Zaharia <matei@eecs.berkeley.edu>
Committed: Sat Nov 9 13:48:00 2013 -0800

----------------------------------------------------------------------
 .../apache/spark/deploy/SparkHadoopUtil.scala   | 21 +++++++++++++++-----
 .../scala/org/apache/spark/rdd/HadoopRDD.scala  |  2 ++
 2 files changed, 18 insertions(+), 5 deletions(-)
----------------------------------------------------------------------



Mime
View raw message