spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject git commit: [SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH
Date Thu, 25 Sep 2014 23:49:19 GMT
Repository: spark
Updated Branches:
  refs/heads/master ff637c938 -> 0dc868e78


[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #2444 from sarutak/slaves-scripts-modification and squashes the following commits:

eff7394 [Kousuke Saruta] Improve the description about Cluster Launch Script in docs/spark-standalone.md
7858225 [Kousuke Saruta] Modified sbin/slaves to use the environment variable "SPARK_SSH_FOREGROUND"
as a flag
53d7121 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into slaves-scripts-modification
e570431 [Kousuke Saruta] Added a description for SPARK_SSH_FOREGROUND variable
7120a0c [Kousuke Saruta] Added a description about default host for sbin/slaves
1bba8a9 [Kousuke Saruta] Added SPARK_SSH_FOREGROUND flag to sbin/slaves
88e2f17 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into slaves-scripts-modification
297e75d [Kousuke Saruta] Modified sbin/slaves not to export HOSTLIST


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0dc868e7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0dc868e7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0dc868e7

Branch: refs/heads/master
Commit: 0dc868e787a3bc69c1b8e90d916a6dcea8dbcd6d
Parents: ff637c9
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Authored: Thu Sep 25 16:49:15 2014 -0700
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Thu Sep 25 16:49:15 2014 -0700

----------------------------------------------------------------------
 .gitignore               |  1 +
 .rat-excludes            |  1 +
 conf/slaves              |  2 --
 conf/slaves.template     |  2 ++
 docs/spark-standalone.md |  7 ++++++-
 sbin/slaves.sh           | 31 ++++++++++++++++++++++---------
 6 files changed, 32 insertions(+), 12 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/0dc868e7/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index 7779980..34939e3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,7 @@ conf/*.cmd
 conf/*.properties
 conf/*.conf
 conf/*.xml
+conf/slaves
 docs/_site
 docs/api
 target/

http://git-wip-us.apache.org/repos/asf/spark/blob/0dc868e7/.rat-excludes
----------------------------------------------------------------------
diff --git a/.rat-excludes b/.rat-excludes
index 9fc99d7..b14ad53 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -19,6 +19,7 @@ log4j.properties
 log4j.properties.template
 metrics.properties.template
 slaves
+slaves.template
 spark-env.sh
 spark-env.cmd
 spark-env.sh.template

http://git-wip-us.apache.org/repos/asf/spark/blob/0dc868e7/conf/slaves
----------------------------------------------------------------------
diff --git a/conf/slaves b/conf/slaves
deleted file mode 100644
index da0a013..0000000
--- a/conf/slaves
+++ /dev/null
@@ -1,2 +0,0 @@
-# A Spark Worker will be started on each of the machines listed below.
-localhost
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/spark/blob/0dc868e7/conf/slaves.template
----------------------------------------------------------------------
diff --git a/conf/slaves.template b/conf/slaves.template
new file mode 100644
index 0000000..da0a013
--- /dev/null
+++ b/conf/slaves.template
@@ -0,0 +1,2 @@
+# A Spark Worker will be started on each of the machines listed below.
+localhost
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/spark/blob/0dc868e7/docs/spark-standalone.md
----------------------------------------------------------------------
diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md
index 29b5491..58103fa 100644
--- a/docs/spark-standalone.md
+++ b/docs/spark-standalone.md
@@ -62,7 +62,12 @@ Finally, the following configuration options can be passed to the master
and wor
 
 # Cluster Launch Scripts
 
-To launch a Spark standalone cluster with the launch scripts, you need to create a file called
`conf/slaves` in your Spark directory, which should contain the hostnames of all the machines
where you would like to start Spark workers, one per line. The master machine must be able
to access each of the slave machines via password-less `ssh` (using a private key). For testing,
you can just put `localhost` in this file.
+To launch a Spark standalone cluster with the launch scripts, you should create a file called
conf/slaves in your Spark directory,
+which must contain the hostnames of all the machines where you intend to start Spark workers,
one per line.
+If conf/slaves does not exist, the launch scripts defaults to a single machine (localhost),
which is useful for testing.
+Note, the master machine accesses each of the worker machines via ssh. By default, ssh is
run in parallel and requires password-less (using a private key) access to be setup.
+If you do not have a password-less setup, you can set the environment variable SPARK_SSH_FOREGROUND
and serially provide a password for each worker.
+
 
 Once you've set up this file, you can launch or stop your cluster with the following shell
scripts, based on Hadoop's deploy scripts, and available in `SPARK_HOME/bin`:
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0dc868e7/sbin/slaves.sh
----------------------------------------------------------------------
diff --git a/sbin/slaves.sh b/sbin/slaves.sh
index 1d4dc5e..cdad47e 100755
--- a/sbin/slaves.sh
+++ b/sbin/slaves.sh
@@ -44,7 +44,9 @@ sbin="`cd "$sbin"; pwd`"
 # If the slaves file is specified in the command line,
 # then it takes precedence over the definition in
 # spark-env.sh. Save it here.
-HOSTLIST="$SPARK_SLAVES"
+if [ -f "$SPARK_SLAVES" ]; then
+  HOSTLIST=`cat "$SPARK_SLAVES"`
+fi
 
 # Check if --config is passed as an argument. It is an optional parameter.
 # Exit if the argument is not a directory.
@@ -67,23 +69,34 @@ fi
 
 if [ "$HOSTLIST" = "" ]; then
   if [ "$SPARK_SLAVES" = "" ]; then
-    export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+    if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+      HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+    else
+      HOSTLIST=localhost
+    fi
   else
-    export HOSTLIST="${SPARK_SLAVES}"
+    HOSTLIST=`cat "${SPARK_SLAVES}"`
   fi
 fi
 
+
+
 # By default disable strict host key checking
 if [ "$SPARK_SSH_OPTS" = "" ]; then
   SPARK_SSH_OPTS="-o StrictHostKeyChecking=no"
 fi
 
-for slave in `cat "$HOSTLIST"|sed  "s/#.*$//;/^$/d"`; do
- ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
-   2>&1 | sed "s/^/$slave: /" &
- if [ "$SPARK_SLAVE_SLEEP" != "" ]; then
-   sleep $SPARK_SLAVE_SLEEP
- fi
+for slave in `echo "$HOSTLIST"|sed  "s/#.*$//;/^$/d"`; do
+  if [ -n "${SPARK_SSH_FOREGROUND}" ]; then
+    ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
+      2>&1 | sed "s/^/$slave: /"
+  else
+    ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
+      2>&1 | sed "s/^/$slave: /" &
+  fi
+  if [ "$SPARK_SLAVE_SLEEP" != "" ]; then
+    sleep $SPARK_SLAVE_SLEEP
+  fi
 done
 
 wait


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message