Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@ignite.apache.org
MIME-Version: 1.0
From: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Date: Wed, 6 Jul 2016 15:20:43 -0400
Message-ID: <CADgKzdxaykd-Bdxnn+9t+RajAZUzLvqCG5LgwcuiaKSj-iQ3+g@mail.gmail.com>
Subject: [Spark-Ignite] How to run exactly one Ignite worker in each Spark
 cluster node.
To: user@ignite.apache.org
Content-Type: multipart/alternative; boundary=001a11412ba899b9f00536fc79a6
archived-at: Wed, 06 Jul 2016 19:21:28 -0000

--001a11412ba899b9f00536fc79a6
Content-Type: text/plain; charset=UTF-8

Hi,

I'm using the Ignite embedded deployment to run an Ignite workload in a
Spark cluster.

In my use case it's required to deploy exactly an Ignite worker for each
node in the Spark cluster. However I haven't found a way to do that.


Take in consideration this scenario: I'm running a 3 nodes Spark cluster on
AWS (1 driver, 2 workers, each node with 3 cores). I would run 2 Ignite
workers, one for each Spark worker.

I'm using the following script:

https://gist.github.com/pditommaso/660cbee09755b2b880099ab3bf2c609a


I've set `spark.executor.instances = 2` in order to deploy two Ignite
workers, indeed in the main log I can read the following:


16/07/06 18:58:02 INFO spark.IgniteContext: Will start Ignite nodes on 2
workers


 However what happens is that Ignite is launched only in one Spark node.


Looking in the same log, the following line seems to suggest the reason:

16/07/06 18:58:05 INFO scheduler.TaskSetManager: Starting task 0.0 in stage
0.0 (TID 0, *ip-10-37-175-68*.eu-west-1.compute.internal, partition
0,PROCESS_LOCAL, 2137 bytes)

16/07/06 18:58:05 INFO scheduler.TaskSetManager: Starting task 1.0 in stage
0.0 (TID 1, *ip-10-37-175-68*.eu-west-1.compute.internal, partition
1,PROCESS_LOCAL, 2194 bytes)


Spark is running two tasks to deploy the Ignite workers, but both of them
in the same node (*ip-10-37-175-68*).

Is there any workaround to avoid this? or more in general, is it possible
to deploy exactly one Ignite worker for each node in the Spark cluster ?


Thanks a lot.

Cheers,
Paolo

--001a11412ba899b9f00536fc79a6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,=C2=A0<div><br></div><div>I&#39;m using the Ignite embe=
dded deployment to run an Ignite workload in a Spark cluster.=C2=A0</div><d=
iv><br></div><div>In my use case it&#39;s required to deploy exactly an Ign=
ite worker for each node in the Spark cluster. However I haven&#39;t found =
a way to do that.=C2=A0</div><div><br></div><div><br></div><div>Take in con=
sideration this scenario: I&#39;m running a 3 nodes Spark cluster on AWS (1=
 driver, 2 workers, each node with 3 cores). I would run 2 Ignite workers, =
one for each Spark worker.=C2=A0</div><div><br></div><div>I&#39;m using the=
 following script: =C2=A0</div><div><br></div><div><a href=3D"https://gist.=
github.com/pditommaso/660cbee09755b2b880099ab3bf2c609a">https://gist.github=
.com/pditommaso/660cbee09755b2b880099ab3bf2c609a</a><br></div><div><br></di=
v><div><br></div><div>I&#39;ve set `spark.executor.instances =3D 2` in orde=
r to deploy two Ignite workers, indeed in the main log I can read the follo=
wing:=C2=A0</div><div><br></div><div><br></div>


<blockquote style=3D"margin:0px 0px 0px 40px;border:none;padding:0px"><div>=
<font face=3D"monospace, monospace">16/07/06 18:58:02 INFO spark.IgniteCont=
ext: Will start Ignite nodes on 2 workers</font></div></blockquote><div><br=
></div><div><br></div><div>=C2=A0However what happens is that Ignite is lau=
nched only in one Spark node.=C2=A0</div><div><br></div><div><br></div><div=
>Looking in the same log, the following line seems to suggest the reason:</=
div><div><br></div><blockquote style=3D"margin:0px 0px 0px 40px;border:none=
;padding:0px"><div><p class=3D""><span class=3D""><font face=3D"monospace, =
monospace">16/07/06 18:58:05 INFO scheduler.TaskSetManager: Starting task 0=
.0 in stage 0.0 (TID 0, <b>ip-10-37-175-68</b>.eu-west-1.compute.internal, =
partition 0,PROCESS_LOCAL, 2137 bytes)</font></span></p></div><div><p class=
=3D""><span class=3D""><font face=3D"monospace, monospace">16/07/06 18:58:0=
5 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, <b>=
ip-10-37-175-68</b>.eu-west-1.compute.internal, partition 1,PROCESS_LOCAL, =
2194 bytes)</font></span></p></div></blockquote><div><br></div><div>Spark i=
s running two tasks to deploy the Ignite workers, but both of them in the s=
ame node (<b style=3D"font-family:monospace,monospace">ip-10-37-175-68</b>)=
.=C2=A0</div><div><br></div><div>Is there any workaround to avoid this? or =
more in general, is it possible to deploy exactly one Ignite worker for eac=
h node in the Spark cluster ? <br><br></div><div><br></div><div>Thanks a lo=
t.=C2=A0</div><div><br></div><div>Cheers,</div><div>Paolo</div><div>=C2=A0<=
/div></div>

--001a11412ba899b9f00536fc79a6--