spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Quanmao Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-17256) spark-submit.cmd cannot work if path has space and cut off double-quoted arguments
Date Fri, 26 Aug 2016 06:30:23 GMT

     [ https://issues.apache.org/jira/browse/SPARK-17256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Quanmao Liu updated SPARK-17256:
--------------------------------
    Description: 
The key problem is  { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept arguments which has
double quotes, even if itself of "spark-xxx.cmd" cannot be just started.

Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc) as following:
 cmd /V /E /C "%~dp0spark-submit2.cmd" %*
Will cut off the argument that double quoted, and not effect if full path of spark-submit.cmd
has space.

Example case :
spark-submit.cmd --jars xxxx  "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table

(1) Cannot just start spark-submit.cmd if it's full path has space, for example  "D:\opengit\spark\bin
- Copy\spark-submit.cmd" :
D:\opengit\spark\bin - Copy>
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table
------- will get error : 
'D:\opengit\spark\bin' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.

(2) spark-submit.cmd can be started but will cut off arguments if path's no space . for example
of above but no space:
D:\opengit\spark\bin>
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table
------- will get error :
'password' is not recognized as an internal or external command,
operable program or batch file.

(1) A solution for this is just removing the "cmd /V /E /C" in the spark/bin/xxx.cmd files
, but this will has possible "pollution" to the environment variables in the same cmd window
(and later derived cmd window like started in it by "start cmd").
But this will solve the space in path problem.

(2) A mitigation is checking before adding double quotes, and added if really need. This just
works well if path no space.  

  was:
The key problem is  { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept arguments has double
quotes, even if itself of "spark-xxx.cmd".

Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc) as following:
 cmd /V /E /C "%~dp0spark-submit2.cmd" %*
Will cut off the argument that double quoted, and not effect if full path of spark-submit.cmd
has space.

Example case :
spark-submit.cmd --jars xxxx  "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table

(1) Cannot just start spark-submit.cmd if it's full path has space, for example  "D:\opengit\spark\bin
- Copy\spark-submit.cmd" :

D:\opengit\spark\bin - Copy>
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table
---- will get erorr : 
'D:\opengit\spark\bin' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.

(2) spark-submit.cmd can be started but will cut off arguments if path's no space . for example
of above but no space:
D:\opengit\spark\bin>
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table

----- will get error :
'password' is not recognized as an internal or external command,
operable program or batch file.

(1) A solution for this is just removing the "cmd /V /E /C" in the spark/bin/xxx.cmd files
, but this will has possible "pollution" to the environment variables in the same cmd window
(and later derived cmd window like started in it by "start cmd").
But this will solve the space in path problem.

(2) A mitigation is checking before adding double quotes, and added if really need. This just
works well if path no space.


> spark-submit.cmd cannot work if path has space and cut off double-quoted arguments
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-17256
>                 URL: https://issues.apache.org/jira/browse/SPARK-17256
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 2.0.0
>         Environment: Windows
>            Reporter: Quanmao Liu
>              Labels: cut-off, double-quotes
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> The key problem is  { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept arguments which
has double quotes, even if itself of "spark-xxx.cmd" cannot be just started.
> Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc) as following:
>  cmd /V /E /C "%~dp0spark-submit2.cmd" %*
> Will cut off the argument that double quoted, and not effect if full path of spark-submit.cmd
has space.
> Example case :
> spark-submit.cmd --jars xxxx  "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table
> (1) Cannot just start spark-submit.cmd if it's full path has space, for example  "D:\opengit\spark\bin
- Copy\spark-submit.cmd" :
> D:\opengit\spark\bin - Copy>
> spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table
> ------- will get error : 
> 'D:\opengit\spark\bin' is not recognized as an internal or external command,
> operable program or batch file.
> 'password' is not recognized as an internal or external command,
> operable program or batch file.
> (2) spark-submit.cmd can be started but will cut off arguments if path's no space . for
example of above but no space:
> D:\opengit\spark\bin>
> spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123"
my_table
> ------- will get error :
> 'password' is not recognized as an internal or external command,
> operable program or batch file.
> (1) A solution for this is just removing the "cmd /V /E /C" in the spark/bin/xxx.cmd
files , but this will has possible "pollution" to the environment variables in the same cmd
window (and later derived cmd window like started in it by "start cmd").
> But this will solve the space in path problem.
> (2) A mitigation is checking before adding double quotes, and added if really need. This
just works well if path no space.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message