The property is deprecated but will still work. Either one is fine.
Launching the job from the namenode is fine .
I brought up a cluster with 2.0.5-alpha and built the latest spark master branch and it runs fine for me. It looks like namenode 2.0.5-alpha won't even start with the defaulFs of file:///. Please make sure your namenode is actually up and running and you are pointing to it because you can run some jobs successfully without it (on a single node cluster), but when you have a multinode cluster here is the error I get when I run without a namenode up and it looks very similar to your error message:
appDiagnostics: Application application_1384876319080_0001 failed 1 times due to AM Container for appattempt_1384876319080_0001_000001 exited with exitCode: -1000 due to: java.io.FileNotFoundException: File file:/home/tgravescs/spark-master/assembly/target/scala-2.9.3/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.0.5-alpha.jar does not exist
When you changed the default fs config did you restart the cluster?
Can you try just running the examples jar:
./spark-class org.apache.spark.deploy.yarn.Client --jar examples/target/scala-2.9.3/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar --class org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers 2 --master-memory 2g --worker-memory 2g --worker-cores 1
On the client side you should see messages like this:
13/11/19 15:41:30 INFO yarn.Client: Uploading file:/home/tgravescs/spark-master/examples/target/scala-2.9.3/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar to hdfs://namenode.host.com:9000/user/tgravescs/.sparkStaging/application_1384874528558_0003/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
13/11/19 15:41:31 INFO yarn.Client: Uploading file:/home/tgravescs/spark-master/assembly/target/scala-2.9.3/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.0.5-alpha.jar to
On Tuesday, November
19, 2013 5:35 AM, guojc <email@example.com> wrote:
Thank you for your response. I have double checked that I had upload both jar in the same folder on hdfs. I think the <name>fs.default.name</name> you pointed out is the old deprecated name for fs.defaultFS config accordiing http://hadoop.apache.org/docs/r2.0.2-alpha/hadoop-project-dist/hadoop-common/DeprecatedProperties.html . Anyway, we have tried both fs.default.name
and fs.defaultFS set to hdfs namenode, and the situation remained same. And we have removed SPARK_HOME env variable on worker node. An additional information might be related is that job submission is done on the same machine of hdfs namenode. But I'm not sure this will cause the problem.