spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: Is spark 1.0.0 "spark-shell --master=yarn" running in yarn-cluster mode or yarn-client mode?
Date Wed, 21 May 2014 18:07:55 GMT
The answer is actually yarn-client. A quick way to find out:

$ bin/spark-shell --master yarn --verbose

>From the system properties you can see spark.master is set to
"yarn-client." From the code, this is because args.deployMode is null, and
so it's not equal to "cluster" and so it falls into the second "if" case
you mentioned:

if (args.deployMode != "cluster" && args.master.startsWith("yarn")) {

  args.master = "yarn-client"

}

2014-05-21 10:57 GMT-07:00 Andrew Lee <alee526@hotmail.com>:

> Does anyone know if:
>
> ./bin/spark-shell --master yarn
>
> is running yarn-cluster or yarn-client by default?
>
> Base on source code:
>
> ./core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
>
>
>     if (args.deployMode == "cluster" && args.master.startsWith("yarn")) {
>
>       args.master = "yarn-cluster"
>
>     }
>
>     if (args.deployMode != "cluster" && args.master.startsWith("yarn")) {
>
>       args.master = "yarn-client"
>
>     }
>
> It looks like the answer is yarn-cluster mode.
>
> I want to confirm this with the community, thanks.
>

Mime
View raw message