ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From what0124 <j.mendoza0...@gmail.com>
Subject Re: Running Spark app in cluster!
Date Fri, 13 Jan 2017 21:07:11 GMT
Sure! I'm trying to set up spark and ignite under CDH for shared deployment
and run this example https://github.com/knoldus/spark-ignite 

What I have done:
1. Download Ignite binaries and set IGNITE_HOME
2. Added library dependencies (ignite-core, ignite-spark and ignite-spring)
and built it using sbt assembly
3. Added JARs to Spark classpath (spark-env.sh)
4. Started Ignite nodes using ./bin/ignite.sh (except master)
4. Submitted Spark job (spark-submit --master yarn --deploy-mode

It successfully creates 1024 partitions and creates cache but when
retrieving RDD the results are not consistent for example:

 val data= Array (1,2,3,4,5,6,7,8,9,10)
 sharedRDD.savePairs(sc.parallelize(data, 10).map(i=> (i, 1)))

val sharedRDD = ic.fromCache[Integer, Integer]("partitioned")
println("The count is:::::::::::: "+sharedRDD.count())

The count at times is 4, other times 10, etc...I don't know if it is some
configuration setting that I'm missing in Cloudera or if there are some
locks needed when reading and writing to cache. Any suggestions would be
appreciated. Thanks!


View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-app-in-cluster-tp10073p10079.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message