spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Kaspar <florian.kas...@onelogic.de>
Subject Re: Programmatically connect to remote YARN in yarn-client mode
Date Wed, 14 Oct 2015 17:29:05 GMT
Thank you, Marcelo,

so it is possible to simply copy the YARN configuration from the remote 
cluster to the local machine (assuming, the local machine can resolve 
the YARN host etc.) and just letting Spark do the rest?
This would actually be great!
Our "local" machine is just another virtual machine running in the same 
environment, connected to the cluster via a virtual network.

Cheers
Florian

Am 14.10.2015 um 19:13 schrieb Marcelo Vanzin:
> On Wed, Oct 14, 2015 at 10:01 AM, Florian Kaspar
> <florian.kaspar@onelogic.de> wrote:
>> we are working on a project running on Spark. Currently we connect to a remote Spark-Cluster
in Standalone mode to obtain the SparkContext using
>>
>> new JavaSparkContext(new SparkConf().setAppName("<AppName>").setMaster("spark://<remoteClusterAddress>:7077"));
>> Can anyone tell me how to create a Spark context programmatically connecting to a
remote YARN cluster?
> You should be able to replace the standalone URL with "yarn-client",
> and it should work, assuming you have the HADOOP_CONF_DIR (or
> YARN_CONF_DIR) env variable pointing at a valid YARN configuration.
>
> Note that if the machine running this code is far from the cluster
> performance might not be that great.
>

-- 
Florian Kaspar

ONE LOGIC

Dr. Hans-Kapfinger-Str. 3, DE 94032 Passau
T +49 851 22590 25
florian.kaspar@onelogic.de
www.onelogic.de


ONE LOGIC GmbH, HRB 7780 Amtsgericht Passau
Geschäftsführung Andreas Böhm, Prof. Dr. Andreas Pfeifer


Mime
View raw message