spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "guoxiaolongzte (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21325) The shell of 'spark-submit' about '--jars' and '--fils', jars and files can be placed on local and hdfs.
Date Thu, 06 Jul 2017 09:23:01 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

guoxiaolongzte updated SPARK-21325:
-----------------------------------
    Description: 
1.My submit way:
spark-submit --class cn.gxl.TestSql{color:red} --jars hdfs://nameservice:/gxl/spark-core_2.11-2.3.0-SNAPSHOT.jar,hdfs://nameservice:/gxl/zookeeper-3.4.6.jar
--files hdfs://nameservice:/gxl/value1.txt,hdfs://nameservice:/gxl/value2.txt{color} hdfs://nameservice:/gxl/spark_2.0.2_project.jar

2.spark-submit description:
--jars JARS {color:red}Comma-separated list of local jars{color} to include on the driver
and executor classpaths.
--files{color:red} FILES Comma-separated list of files{color} to be placed in the working
directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).

3.Problem Description:
{color:red}*jars and files Not only can be placed on local but also can be placed on hdfs.

The description of '' - jars '', that can only be placed on local.This is wrong

The description of '--files' is not clear that can be placed locally or hdfs.This is blurry.
Not conducive to the developer to understand and use.*{color}

So, this is an optimization feature that deserves to be modified.

  was:
1.My submit way:
spark-submit --class cn.gxl.TestSql --jars hdfs://nameservice:/gxl/spark-core_2.11-2.3.0-SNAPSHOT.jar,hdfs://nameservice:/gxl/zookeeper-3.4.6.jar
--files hdfs://nameservice:/gxl/value1.txt,hdfs://nameservice:/gxl/value2.txt hdfs://nameservice:/gxl/spark_2.0.2_project.jar
2.spark-submit description:
--jars JARS Comma-separated list of local jars to include on the driver and executor classpaths.
--files FILES Comma-separated list of files to be placed in the working directory of each
executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
3.Problem Description:
jars and files Not only can be placed on local but also can be placed on hdfs.
The description of '' - jars '', that can only be placed on local.This is wrong
The description of '--files' is not clear that can be placed locally or hdfs.This is blurry.
Not conducive to the developer to understand and use.
So, this is an optimization feature that deserves to be modified.


> The shell of 'spark-submit' about '--jars' and '--fils', jars and files can be placed
on local and hdfs.
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-21325
>                 URL: https://issues.apache.org/jira/browse/SPARK-21325
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: guoxiaolongzte
>            Priority: Minor
>
> 1.My submit way:
> spark-submit --class cn.gxl.TestSql{color:red} --jars hdfs://nameservice:/gxl/spark-core_2.11-2.3.0-SNAPSHOT.jar,hdfs://nameservice:/gxl/zookeeper-3.4.6.jar
--files hdfs://nameservice:/gxl/value1.txt,hdfs://nameservice:/gxl/value2.txt{color} hdfs://nameservice:/gxl/spark_2.0.2_project.jar
> 2.spark-submit description:
> --jars JARS {color:red}Comma-separated list of local jars{color} to include on the driver
and executor classpaths.
> --files{color:red} FILES Comma-separated list of files{color} to be placed in the working
directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
> 3.Problem Description:
> {color:red}*jars and files Not only can be placed on local but also can be placed on
hdfs.
> The description of '' - jars '', that can only be placed on local.This is wrong
> The description of '--files' is not clear that can be placed locally or hdfs.This is
blurry. Not conducive to the developer to understand and use.*{color}
> So, this is an optimization feature that deserves to be modified.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message