spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rex Xiong <bycha...@gmail.com>
Subject Issue of jar dependency in yarn-cluster mode
Date Fri, 16 Oct 2015 14:57:20 GMT
Hi folks,

In my spark application, executor task depends on snakeyaml-1.10.jar
I build it with Maven and it works fine:
spark-submit --master local --jars d:\snakeyaml-1.10.jar ...........

But when I try to run it in yarn, I have issue, it seems spark executor
cannot find the jar file:
spark-submit --master yarn-cluster --jars hdfs://....../snakeyaml-1.10.jar
  ..........

java.lang.NoSuchMethodError:
org.yaml.snakeyaml.Yaml.<init>(Lorg/yaml/snakeyaml/constructor/BaseConstructor;)V

I check one executor container folder, snakeyaml-1.10.jar has been
successfully downloaded, and in spark driver page, in environment tab,
spark.yarn.secondary.jars
also contains snakeyaml-1.10.jar

I have no idea why it doesn't work. Could some one help to take a look?

Thanks

Mime
View raw message