spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: Spark on Yarn - How to configure
Date Sat, 20 Jun 2015 01:50:53 GMT
Hi Ashish,

For Spark on YARN, you actually only need the Spark files on one machine -
the submission client. This machine could even live outside of the cluster.
Then all you need to do is point YARN_CONF_DIR to the directory containing
your hadoop configuration files (e.g. yarn-site.xml) on that machine. All
the jars will be automatically distributed to the nodes in the cluster
accordingly.

-Andrew

2015-06-19 12:35 GMT-07:00 Ashish Soni <asoni.learn@gmail.com>:

> Can some one please let me know what all i need to configure to have Spark
> run using Yarn ,
>
> There is lot of documentation but none of it says how and what all files
> needs to be changed
>
> Let say i have 4 node for Spark - SparkMaster , SparkSlave1 , SparkSlave2
> , SparkSlave3
>
> Now in which node which files needs to changed to make sure my master node
> is SparkMaster and slave nodes are 1,2,3 and how to tell / configure Yarn
>
> Ashish
>

Mime
View raw message