spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1395) Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR
Date Wed, 02 Apr 2014 20:25:17 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin updated SPARK-1395:
----------------------------------

    Summary: Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR  (was: Cannoit
launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR)

> Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR
> --------------------------------------------------------------------
>
>                 Key: SPARK-1395
>                 URL: https://issues.apache.org/jira/browse/SPARK-1395
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Marcelo Vanzin
>
> If you define SPARK_JAR and friends to use "local:" URIs, you cannot submit a job on
a Yarn cluster. e.g., I have:
> SPARK_JAR=local:/tmp/spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar
> SPARK_YARN_APP_JAR=local:/tmp/spark-examples-assembly-1.0.0-SNAPSHOT.jar
> And running SparkPi using bin/run-example yields this:
> {noformat}
> 14/04/02 13:23:33 INFO yarn.Client: Preparing Local resources
> Exception in thread "main" java.io.IOException: No FileSystem for scheme: local
>         at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2385)
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
>         at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
>         at org.apache.spark.deploy.yarn.ClientBase$class.org$apache$spark$deploy$yarn$ClientBase$$copyRemoteFile(ClientBase.scala:156)
>         at org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$3.apply(ClientBase.scala:217)
>         at org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$3.apply(ClientBase.scala:212)
>         at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>         at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>         at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
>         at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>         at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>         at org.apache.spark.deploy.yarn.ClientBase$class.prepareLocalResources(ClientBase.scala:212)
>         at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:41)
>         at org.apache.spark.deploy.yarn.Client.runApp(Client.scala:76)
>         at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:81)
>         at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:129)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:226)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:96)
>         at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
>         at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message