spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tristan Nixon <st...@memeticlabs.org>
Subject Re: How to distribute dependent files (.so , jar ) across spark worker nodes
Date Fri, 11 Mar 2016 22:35:00 GMT
I recommend you package all your dependencies (jars, .so’s, etc.) into a single uber-jar
and then submit that. It’s much more convenient than trying to manage including everything
in the --jars arg of spark-submit. If you build with maven than the shade plugin will do this
for you nicely:
https://maven.apache.org/plugins/maven-shade-plugin/

> On Mar 11, 2016, at 2:05 PM, Jacek Laskowski <jacek@japila.pl> wrote:
> 
> Hi,
> 
> For jars use spark-submit --jars. Dunno about so's. Could that work through jars?
> 
> Jacek
> 
> 11.03.2016 8:07 PM "prateek arora" <prateek.arora2k6@gmail.com <mailto:prateek.arora2k6@gmail.com>>
napisał(a):
> Hi
> 
> I have multiple node cluster and my spark jobs depend on a native
> library (.so files) and some jar files.
> 
> Can some one please explain what are the best ways to distribute dependent
> files across nodes?
> 
> right now i copied  dependent files in all nodes using chef tool .
> 
> Regards
> Prateek
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-distribute-dependent-files-so-jar-across-spark-worker-nodes-tp26464.html
<http://apache-spark-user-list.1001560.n3.nabble.com/How-to-distribute-dependent-files-so-jar-across-spark-worker-nodes-tp26464.html>
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <mailto:user-unsubscribe@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org <mailto:user-help@spark.apache.org>
> 


Mime
View raw message