ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Di Tommaso <paolo.ditomm...@gmail.com>
Subject [Spark-Ignite] How to run exactly one Ignite worker in each Spark cluster node.
Date Wed, 06 Jul 2016 19:20:43 GMT
Hi,

I'm using the Ignite embedded deployment to run an Ignite workload in a
Spark cluster.

In my use case it's required to deploy exactly an Ignite worker for each
node in the Spark cluster. However I haven't found a way to do that.


Take in consideration this scenario: I'm running a 3 nodes Spark cluster on
AWS (1 driver, 2 workers, each node with 3 cores). I would run 2 Ignite
workers, one for each Spark worker.

I'm using the following script:

https://gist.github.com/pditommaso/660cbee09755b2b880099ab3bf2c609a


I've set `spark.executor.instances = 2` in order to deploy two Ignite
workers, indeed in the main log I can read the following:


16/07/06 18:58:02 INFO spark.IgniteContext: Will start Ignite nodes on 2
workers



 However what happens is that Ignite is launched only in one Spark node.


Looking in the same log, the following line seems to suggest the reason:

16/07/06 18:58:05 INFO scheduler.TaskSetManager: Starting task 0.0 in stage
0.0 (TID 0, *ip-10-37-175-68*.eu-west-1.compute.internal, partition
0,PROCESS_LOCAL, 2137 bytes)

16/07/06 18:58:05 INFO scheduler.TaskSetManager: Starting task 1.0 in stage
0.0 (TID 1, *ip-10-37-175-68*.eu-west-1.compute.internal, partition
1,PROCESS_LOCAL, 2194 bytes)


Spark is running two tasks to deploy the Ignite workers, but both of them
in the same node (*ip-10-37-175-68*).

Is there any workaround to avoid this? or more in general, is it possible
to deploy exactly one Ignite worker for each node in the Spark cluster ?


Thanks a lot.

Cheers,
Paolo

Mime
View raw message