ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From percent620 <percent...@163.com>
Subject Ignite for Spark on YARN Deployment
Date Mon, 15 Aug 2016 01:31:11 GMT
My detailed issues as below this link, please refer to this link
http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tt6910.html


My question:
1) What ports should be opened on driver machine or yarn machine?(There
issues were happned on our production env)

2) But in my local env(3 spark cluster[1 master, 2 workers] and 1 yarn
cluster) and there is no issues as i manually closed my firewall, then fixed
this issue.

I faced another issues as below
========================
1)
val igniteContext = new IgniteContext[Integer,Integer](sc, () => new
IgniteConfiguration()) 
    val sharedRDD = igniteContext.fromCache("sharedBaselineCacheRDD") 
    val initalRDD = sc.parallelize(1 to 100000,10).map(i => (new Integer(i),
new Integer(i))) 
    println("initalRDD.couner=/. " + initalRDD.count() +"\tpartition=> " +
initalRDD.partitions.size) 

    sharedRDD.savePairs(initalRDD, true) 
    println("=====>totalcounter" + sharedRDD.count + "\t paris => " +
sharedRDD.partitions.size) 
    println("=====>" + sharedRDD.filter(_._2 > 50000).count) 


2) submit a jar to yarn cluster as below command
/u01/spark-1.6.0-hive/bin/spark-submit --driver-memory 4G --class
com.ValidSparkCache --master yarn --executor-cores 5 --executor-memory 1000m
--num-executors 10 --conf spark.rdd.compress=false --conf
spark.shuffle.compress=false --conf spark.broadcast.compress=false
/u01/xxx/sparkignite.jar -quesize 10 -batchSize 5000 -writethread 30
-runningSeconds 20

3) final result should be 50000, but actullay result is 40000,3000(ramdom)

final result is 40000 NOT 50000, this is happened also on embed integration
with spark.



--
View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp10686.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Mime
View raw message