ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From percent620 <percent...@163.com>
Subject Ignite for Spark on YARN Deployment
Date Mon, 15 Aug 2016 01:31:11 GMT
My detailed issues as below this link, please refer to this link

My question:
1) What ports should be opened on driver machine or yarn machine?(There
issues were happned on our production env)

2) But in my local env(3 spark cluster[1 master, 2 workers] and 1 yarn
cluster) and there is no issues as i manually closed my firewall, then fixed
this issue.

I faced another issues as below
val igniteContext = new IgniteContext[Integer,Integer](sc, () => new
    val sharedRDD = igniteContext.fromCache("sharedBaselineCacheRDD") 
    val initalRDD = sc.parallelize(1 to 100000,10).map(i => (new Integer(i),
new Integer(i))) 
    println("initalRDD.couner=/. " + initalRDD.count() +"\tpartition=> " +

    sharedRDD.savePairs(initalRDD, true) 
    println("=====>totalcounter" + sharedRDD.count + "\t paris => " +
    println("=====>" + sharedRDD.filter(_._2 > 50000).count) 

2) submit a jar to yarn cluster as below command
/u01/spark-1.6.0-hive/bin/spark-submit --driver-memory 4G --class
com.ValidSparkCache --master yarn --executor-cores 5 --executor-memory 1000m
--num-executors 10 --conf spark.rdd.compress=false --conf
spark.shuffle.compress=false --conf spark.broadcast.compress=false
/u01/xxx/sparkignite.jar -quesize 10 -batchSize 5000 -writethread 30
-runningSeconds 20

3) final result should be 50000, but actullay result is 40000,3000(ramdom)

final result is 40000 NOT 50000, this is happened also on embed integration
with spark.

View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp10686.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

View raw message