spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carsten Blank (JIRA)" <>
Subject [jira] [Commented] (SPARK-5754) Spark AM not launching on Windows
Date Fri, 31 Jul 2015 15:39:05 GMT


Carsten Blank commented on SPARK-5754:

Thanks, I will put in more time and come up with a suggestion. 
Catch ya later

> Spark AM not launching on Windows
> ---------------------------------
>                 Key: SPARK-5754
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Windows, YARN
>    Affects Versions: 1.1.1, 1.2.0
>         Environment: Windows Server 2012, Hadoop 2.4.1.
>            Reporter: Inigo
> I'm trying to run Spark Pi on a YARN cluster running on Windows and the AM container
fails to start. The problem seems to be in the generation of the YARN command which adds single
quotes (') surrounding some of the java options. In particular, the part of the code that
is adding those is the escapeForShell function in YarnSparkHadoopUtil. Apparently, Windows
does not like the quotes for these options. Here is an example of the command that the container
tries to execute:
> @call %JAVA_HOME%/bin/java -server -Xmx512m '-Dspark.yarn.secondary.jars='
'' '-Dspark.master=yarn-cluster' org.apache.spark.deploy.yarn.ApplicationMaster
--class 'org.apache.spark.examples.SparkPi' --jar  'file:/D:/data/spark-1.1.1-bin-hadoop2.4/bin/../lib/spark-examples-1.1.1-hadoop2.4.0.jar'
 --executor-memory 1024 --executor-cores 1 --num-executors 2
> Once I transform it into:
> @call %JAVA_HOME%/bin/java -server -Xmx512m -Dspark.yarn.secondary.jars= -Dspark.master=yarn-cluster org.apache.spark.deploy.yarn.ApplicationMaster
--class 'org.apache.spark.examples.SparkPi' --jar  'file:/D:/data/spark-1.1.1-bin-hadoop2.4/bin/../lib/spark-examples-1.1.1-hadoop2.4.0.jar'
 --executor-memory 1024 --executor-cores 1 --num-executors 2
> Everything seems to start.
> How should I deal with this? Creating a separate function like escapeForShell for Windows
and call it whenever I detect this is for Windows? Or should I add some sanity check on YARN?
> I checked a little and there seems to be people that is able to run Spark on YARN on
Windows, so it might be something else. I didn't find anything related on Jira either.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message