spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rex Xiong <>
Subject Issue of jar dependency in yarn-cluster mode
Date Fri, 16 Oct 2015 14:57:20 GMT
Hi folks,

In my spark application, executor task depends on snakeyaml-1.10.jar
I build it with Maven and it works fine:
spark-submit --master local --jars d:\snakeyaml-1.10.jar ...........

But when I try to run it in yarn, I have issue, it seems spark executor
cannot find the jar file:
spark-submit --master yarn-cluster --jars hdfs://....../snakeyaml-1.10.jar


I check one executor container folder, snakeyaml-1.10.jar has been
successfully downloaded, and in spark driver page, in environment tab,
also contains snakeyaml-1.10.jar

I have no idea why it doesn't work. Could some one help to take a look?


View raw message