spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Kaspar <florian.kas...@onelogic.de>
Subject Programmatically connect to remote YARN in yarn-client mode
Date Wed, 14 Oct 2015 17:01:54 GMT
Hey everyone,

we are working on a project running on Spark. Currently we connect to a 
remote Spark-Cluster in Standalone mode to obtain the SparkContext using

new JavaSparkContext(new SparkConf().setAppName("<AppName>").setMaster("spark://<remoteClusterAddress>:7077"));

Currently, we try to connect to a remote (!) YARN cluster instead. This 
should also happen programmatically. We use the Spark context for the 
whole lifetime of a Web-Application.
spark-submit is essentially no good option for us because we want to 
have a local Spark Driver connecting to a remote cluster with the driver 
interacting with other locally deployed modules.
Can anyone tell me how to create a Spark context programmatically 
connecting to a remote YARN cluster?
Tutorials online seem to have the precondition that the current 
application runs inside the cluster but not remotely.
Thank you in advance for your support!

Kind regards
Florian

-- 
Florian Kaspar

ONE LOGIC

Dr. Hans-Kapfinger-Str. 3, DE 94032 Passau
T +49 851 22590 25
florian.kaspar@onelogic.de
www.onelogic.de


ONE LOGIC GmbH, HRB 7780 Amtsgericht Passau
Geschäftsführung Andreas Böhm, Prof. Dr. Andreas Pfeifer


Mime
View raw message