spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vincent Sun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20016) SparkLauncher submit job failed after setConf with special charaters under windows
Date Mon, 20 Mar 2017 14:06:41 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932682#comment-15932682
] 

Vincent Sun commented on SPARK-20016:
-------------------------------------

I don't know too much about that. But from my opinion, cmd will treat the parameters in double
quotes as a file name or path. It is used to resolve the escaping white space and special
characters. So for other kind of parameters, should use single quote or not use anything.

> SparkLauncher submit job failed after setConf with special charaters under windows
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-20016
>                 URL: https://issues.apache.org/jira/browse/SPARK-20016
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API
>    Affects Versions: 2.0.0
>         Environment: windows 7, 8, 10, 2008, 2008R2, etc.
>            Reporter: Vincent Sun
>
> I am using sparkLauncher JAVA API to submit job to a remote spark cluster master.  Codes
looks like follow:
> /*
> * launch Job
> */
> public static void launch() throws Exception {
>     SparkLauncher spark = new SparkLauncher();      spark.setAppName("sparkdemo").setAppResource("hdfs://10.250.1.121:9000/application.jar").setMainClass("test.Application");
>     spark.setMaster(spark://10.250.1.120:6066);
>     spark.setDeployMode("cluster");
>     spark.setConf("spark.executor.cores","2") 
>     spark.setConf("spark.executor.memory","8G") 
>     spark.startApplication(new MyAppListener(job.getAppName()));
>   }
> It works fine under Linux/CentOS, but failed on my own desktop which is a windows 8 OS.
It will throw out error:
> <b>[launcher-proc-1] The filename, directory name, or volume label syntax is incorrect.</b>
> The finial command I caught is this:
> spark-submit.cmd  --master spark://10.250.1.120:6066 --deploy-mode cluster --name sparkdemo
--conf "spark.executor.memory=8G" --conf "spark.executor.cores=2"  --class test.Application
hdfs://10.250.1.121:9000/application.jar
> The quote on spark.executor.memory=8G and spark.executor.cores=2 cause the exception.
> After debug into the source code I found the reason is at:
> quoteForBatchScript method of CommandBuilderUtils class
> It will add quotes while there is '=' or some other kinds of special characters under
windows system. Here is the source codes:
> static String quoteForBatchScript(String arg) {
>     boolean needsQuotes = false;
>     for (int i = 0; i < arg.length(); i++) {
>       int c = arg.codePointAt(i);
>       if (Character.isWhitespace(c) || c == '"' || c == '=' || c == ',' || c == ';')
{
>         needsQuotes = true;
>         break;
>       }
>     }
>     if (!needsQuotes) {
>       return arg;
>     }
>     StringBuilder quoted = new StringBuilder();
>     quoted.append("\"");
>     for (int i = 0; i < arg.length(); i++) {
>       int cp = arg.codePointAt(i);
>       switch (cp) {
>       case '"':
>         quoted.append('"');
>         break;
>       default:
>         break;
>       }
>       quoted.appendCodePoint(cp);
>     }
>     if (arg.codePointAt(arg.length() - 1) == '\\') {
>       quoted.append("\\");
>     }
>     quoted.append("\"");
>     return quoted.toString();
>   }



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message