spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode.
Date Wed, 16 Dec 2015 02:20:12 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.6 9e4ac5645 -> 2c324d35a


[SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode.

Adding more documentation about submitting jobs with mesos cluster mode.

Author: Timothy Chen <tnachen@gmail.com>

Closes #10086 from tnachen/mesos_supervise_docs.

(cherry picked from commit c2de99a7c3a52b0da96517c7056d2733ef45495f)
Signed-off-by: Andrew Or <andrew@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2c324d35
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2c324d35
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2c324d35

Branch: refs/heads/branch-1.6
Commit: 2c324d35a698b353c2193e2f9bd8ba08c741c548
Parents: 9e4ac56
Author: Timothy Chen <tnachen@gmail.com>
Authored: Tue Dec 15 18:20:00 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Tue Dec 15 18:20:09 2015 -0800

----------------------------------------------------------------------
 docs/running-on-mesos.md        | 26 +++++++++++++++++++++-----
 docs/submitting-applications.md | 15 ++++++++++++++-
 2 files changed, 35 insertions(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/2c324d35/docs/running-on-mesos.md
----------------------------------------------------------------------
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md
index a197d0e..3193e17 100644
--- a/docs/running-on-mesos.md
+++ b/docs/running-on-mesos.md
@@ -150,14 +150,30 @@ it does not need to be redundantly passed in as a system property.
 Spark on Mesos also supports cluster mode, where the driver is launched in the cluster and
the client
 can find the results of the driver from the Mesos Web UI.
 
-To use cluster mode, you must start the MesosClusterDispatcher in your cluster via the `sbin/start-mesos-dispatcher.sh`
script,
-passing in the Mesos master url (e.g: mesos://host:5050).
+To use cluster mode, you must start the `MesosClusterDispatcher` in your cluster via the
`sbin/start-mesos-dispatcher.sh` script,
+passing in the Mesos master URL (e.g: mesos://host:5050). This starts the `MesosClusterDispatcher`
as a daemon running on the host.
 
-From the client, you can submit a job to Mesos cluster by running `spark-submit` and specifying
the master url
-to the url of the MesosClusterDispatcher (e.g: mesos://dispatcher:7077). You can view driver
statuses on the
+If you like to run the `MesosClusterDispatcher` with Marathon, you need to run the `MesosClusterDispatcher`
in the foreground (i.e: `bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher`).
+
+From the client, you can submit a job to Mesos cluster by running `spark-submit` and specifying
the master URL
+to the URL of the `MesosClusterDispatcher` (e.g: mesos://dispatcher:7077). You can view driver
statuses on the
 Spark cluster Web UI.
 
-Note that jars or python files that are passed to spark-submit should be URIs reachable by
Mesos slaves.
+For example:
+{% highlight bash %}
+./bin/spark-submit \
+  --class org.apache.spark.examples.SparkPi \
+  --master mesos://207.184.161.138:7077 \
+  --deploy-mode cluster
+  --supervise
+  --executor-memory 20G \
+  --total-executor-cores 100 \
+  http://path/to/examples.jar \
+  1000
+{% endhighlight %}
+
+
+Note that jars or python files that are passed to spark-submit should be URIs reachable by
Mesos slaves, as the Spark driver doesn't automatically upload local jars.
 
 # Mesos Run Modes
 

http://git-wip-us.apache.org/repos/asf/spark/blob/2c324d35/docs/submitting-applications.md
----------------------------------------------------------------------
diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md
index ac2a14e..acbb0f2 100644
--- a/docs/submitting-applications.md
+++ b/docs/submitting-applications.md
@@ -115,6 +115,18 @@ export HADOOP_CONF_DIR=XXX
   --master spark://207.184.161.138:7077 \
   examples/src/main/python/pi.py \
   1000
+
+# Run on a Mesos cluster in cluster deploy mode with supervise
+./bin/spark-submit \
+  --class org.apache.spark.examples.SparkPi \
+  --master mesos://207.184.161.138:7077 \
+  --deploy-mode cluster
+  --supervise
+  --executor-memory 20G \
+  --total-executor-cores 100 \
+  http://path/to/examples.jar \
+  1000
+
 {% endhighlight %}
 
 # Master URLs
@@ -132,9 +144,10 @@ The master URL passed to Spark can be in one of the following formats:
 <tr><td> <code>mesos://HOST:PORT</code> </td><td> Connect
to the given <a href="running-on-mesos.html">Mesos</a> cluster.
         The port must be whichever one your is configured to use, which is 5050 by default.
         Or, for a Mesos cluster using ZooKeeper, use <code>mesos://zk://...</code>.
+        To submit with <code>--deploy-mode cluster</code>, the HOST:PORT should
be configured to connect to the <a href="running-on-mesos.html#cluster-mode">MesosClusterDispatcher</a>.
 </td></tr>
 <tr><td> <code>yarn</code> </td><td> Connect to a <a
href="running-on-yarn.html"> YARN </a> cluster in
-        <code>client</code> or <code>cluster</code> mode depending
on the value of <code>--deploy-mode</code>. 
+        <code>client</code> or <code>cluster</code> mode depending
on the value of <code>--deploy-mode</code>.
         The cluster location will be found based on the <code>HADOOP_CONF_DIR</code>
or <code>YARN_CONF_DIR</code> variable.
 </td></tr>
 <tr><td> <code>yarn-client</code> </td><td> Equivalent
to <code>yarn</code> with <code>--deploy-mode client</code>,


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message