Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 6 Feb 2017 10:42:42 +0000 (UTC)
From: "wangqiaoshi (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12873072.1442428877000.4550.1486377762153@Atlassian.JIRA>
In-Reply-To: <JIRA.12873072.1442428877000@Atlassian.JIRA>
References: <JIRA.12873072.1442428877000@Atlassian.JIRA> <JIRA.12873072.1442428877340@jira-lw-us.apache.org>
Subject: [jira] [Commented] (SPARK-10643) Support remote application
 download in client mode spark submit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 06 Feb 2017 10:42:49 -0000


    [ https://issues.apache.org/jira/browse/SPARK-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853817#comment-15853817 ] 

wangqiaoshi commented on SPARK-10643:
-------------------------------------

+1.
i think it would be useful when  use azkaban in mutil-executor mode,i expect get execution-jar from hdfs but from mysql.

> Support remote application download in client mode spark submit
> ---------------------------------------------------------------
>
>                 Key: SPARK-10643
>                 URL: https://issues.apache.org/jira/browse/SPARK-10643
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Submit
>            Reporter: Alan Braithwaite
>            Priority: Minor
>
> When using mesos with docker and marathon, it would be nice to be able to make spark-submit deployable on marathon and have that download a jar from HDFS instead of having to package the jar with the docker.
> {code}
> $ docker run -it docker.example.com/spark:latest /usr/local/spark/bin/spark-submit  --class com.example.spark.streaming.EventHandler hdfs://hdfs/tmp/application.jar 
> Warning: Skip remote jar hdfs://hdfs/tmp/application.jar.
> java.lang.ClassNotFoundException: com.example.spark.streaming.EventHandler
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:348)
>         at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
>         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:639)
>         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
>         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
> Although I'm aware that we can run in cluster mode with mesos, we've already built some nice tools surrounding marathon for logging and monitoring.
> Code in question:
> https://github.com/apache/spark/blob/132718ad7f387e1002b708b19e471d9cd907e105/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L723-L736


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org