ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From roshan joe <impdocs2...@gmail.com>
Subject Ignite with Spark on Yarn
Date Fri, 03 Nov 2017 13:01:07 GMT
Below are the steps I followed for ignite with spark on Yarn but I am stuck
after spending almost all day on this.

   - copied apache-ignite-fabric-2.1.0-bin (tried 2.3.0 initially but the
   it didnt work) to all the 4 slave nodes of Spark cluster and started ignite
   with ./bin/ignite.sh. It shows as given below,  after starting

*Topology snapshot [ver=10, servers=4, clients=0, CPUs=32, heap=4.0GB]*

   - updated the spark-env.sh on all slave nodes with IGNITE_HOME and class
   paths as below



for file in ${IGNITE_HOME}/libs/*


    if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ];




export spark.driver.extraClassPath=$IGNITE_LIBS

export spark.executor.extraClassPath=$IGNITE_LIBS

   - updated the default config.xml as attached with the static ips
   - On the Spark Master node (Only), added the  - <property name=
   "clientMode" value="true"/>

In the simple test code, i tried 2 things, both without success:

1. Added config/default-config.xml directly under the project and added the
below snippet.

val CONFIG = "config/default-config.xml"
val igniteContext = new IgniteContext(sparkContext, CONFIG, true)
val sharedRDD: IgniteRDD[Int, Int] = igniteContext.fromCache[Int,
sharedRDD.savePairs(sparkContext.parallelize(1 to 100000, 10).map(i => (i, i)))
sharedRDD.mapValues(x => (x * x))

During execution, it complains that - class
Spring XML configuration path is invalid: config/default-config.xml. Note
that this path should be either absolute or a relative local file system
path, relative to META-INF in classpath or valid URL to IGNITE_HOME.

2. Commented the above snippet and used the below:

val ic = new IgniteContext(sparkContext, () => new IgniteConfiguration())
val sharedRDD: IgniteRDD[Int, Int] = ic.fromCache("partitioned")
sharedRDD.savePairs(sparkContext.parallelize(1 to 100000, 10).map(i => (i, i)))

Now it complains that "it cannot find any IPs from multicast or IPFinder.

Not sure what am i doing not right. I also have a cluster.properties file
but I am not sure if it is needed for the yarn setup and I am not
explicitly calling it anywhere. Thank you.

View raw message