Mailing-List: contact user-help@storm.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAO5KYW9bpD-B1hNnUJ25FVGrt1rK7iwrv8wOH1aC=YZmvZe2-g@mail.gmail.com>
References: 
 <CAOy42CuFsACNmsfDeL9Y9sy9sOPdZYQRO2ZeyQRg9Xx_pw+usg@mail.gmail.com>
	<1467130598.2940616.1454357692402.JavaMail.yahoo@mail.yahoo.com>
	<CAO5KYW9bpD-B1hNnUJ25FVGrt1rK7iwrv8wOH1aC=YZmvZe2-g@mail.gmail.com>
Date: Thu, 4 Feb 2016 12:44:43 +0200
Message-ID: 
 <CAOy42CuDgW-UX447FDJx5LQ99C0gpyvTe3N0r0t8X93twjV_Cg@mail.gmail.com>
Subject: Re: running multiple topologies in same cluster
From: Spico Florin <spicoflorin@gmail.com>
To: user@storm.apache.org
Cc: Bobby Evans <evans@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=001a11c3b1da21e75c052aef6c9d

--001a11c3b1da21e75c052aef6c9d
Content-Type: text/plain; charset=UTF-8

Hello!
Thank you all for your answers! I guess I'll wait for adding in support for
resource aware scheduling in a multi-tenant stand alone storm cluster.
Regards,
 Florin

On Tue, Feb 2, 2016 at 1:00 AM, Erik Weathers <eweathers@groupon.com> wrote:

> hi Spico,
>
> As Bobby said, native Storm is going to have better support for this soon.
>
> FWIW, there is also the storm-on-mesos project, which we've been running
> on for almost 2 years at Groupon.
>
>    - https://github.com/mesos/storm
>
> Caveats:
>
>    - storm's logviewer is unsupported
>    - scheduling can be suboptimal, causing topologies of different "size"
>    (resource requirements) to starve each other
>       - side effect of needing to dynamically calculate storm "slots"
>       from mesos resource offers, my team has a framework change we are testing
>       that will improve that behavior.
>    - it's relatively complex to operate since there are so many different
>    moving parts / components
>       - mesos-master
>       - mesos-slave/agent (it's being renamed)
>       - storm nimbus (MesosNimbus -> the mesos scheduler)
>       - storm supervisor (MesosSupervisor -> the mesos executor)
>       - storm worker (the mesos task)
>       - ZooKeeper
>    - it probably won't work nicely with all the fancy security stuff that
>    has been added to Storm in 0.10.0+
>
> - Erik
>
> On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans <evans@yahoo-inc.com> wrote:
>
>> We are currently adding in support for resource aware scheduling in a
>> multi-tenant stand alone storm cluster.  It is still alpha quality but we
>> plan on getting it into production at Yahoo this quarter.  If you can wait
>> that would be the preferred way I see to support your use case.
>>
>> - Bobby
>>
>>
>> On Monday, February 1, 2016 12:16 PM, Spico Florin <spicoflorin@gmail.com>
>> wrote:
>>
>>
>> Hello!
>> I have an use case where we have to deploy many tpologies in a storm
>> cluster.
>> 1.we would like to know if running these topologies in combination with
>> apache slider over yarn would bring us some benefits in terms of resource
>> consumption?
>> 2. in such cases (running many topolgies aprox 60) what are the best
>> practices on how to run them over a cluster with a smart load balanced
>> hardware resources consumption (cpu, ram)?
>> I look forward for your answers.
>> Regards,
>> Florin
>>
>>
>>
>>
>

--001a11c3b1da21e75c052aef6c9d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hello!</div>Thank you all for your answers! I guess I=
&#39;ll wait for=C2=A0<span style=3D"color:rgb(0,0,0);font-family:Helvetica=
Neue-Light,&#39;Helvetica Neue Light&#39;,&#39;Helvetica Neue&#39;,Helvetic=
a,Arial,&#39;Lucida Grande&#39;,sans-serif;font-size:16px">adding in suppor=
t for resource aware scheduling in a multi-tenant=C2=A0</span><span style=
=3D"color:rgb(0,0,0);font-family:HelveticaNeue-Light,&#39;Helvetica Neue Li=
ght&#39;,&#39;Helvetica Neue&#39;,Helvetica,Arial,&#39;Lucida Grande&#39;,s=
ans-serif;font-size:16px">stand alone storm cluster.</span><div><span style=
=3D"color:rgb(0,0,0);font-family:HelveticaNeue-Light,&#39;Helvetica Neue Li=
ght&#39;,&#39;Helvetica Neue&#39;,Helvetica,Arial,&#39;Lucida Grande&#39;,s=
ans-serif;font-size:16px">Regards,</span></div><div><span style=3D"color:rg=
b(0,0,0);font-family:HelveticaNeue-Light,&#39;Helvetica Neue Light&#39;,=
9;Helvetica Neue&#39;,Helvetica,Arial,&#39;Lucida Grande&#39;,sans-serif;fo=
nt-size:16px">=C2=A0Florin</span></div></div><div class=3D"gmail_extra"><br=
><div class=3D"gmail_quote">On Tue, Feb 2, 2016 at 1:00 AM, Erik Weathers <=
span dir=3D"ltr">&lt;<a href=3D"mailto:eweathers@groupon.com" target=3D"_bl=
ank">eweathers@groupon.com</a>&gt;</span> wrote:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex"><div dir=3D"ltr">hi Spico,<div><br></div><div>As Bobby said, native=
 Storm is going to have better support for this soon.</div><div><br></div><=
div>FWIW, there is also the storm-on-mesos project, which we&#39;ve been ru=
nning on for almost 2 years at Groupon.<div><ul><li><a href=3D"https://gith=
ub.com/mesos/storm" target=3D"_blank">https://github.com/mesos/storm</a></l=
i></ul></div><div><div>Caveats:</div><div><ul><li>storm&#39;s logviewer is =
unsupported</li><li>scheduling can be suboptimal, causing topologies of dif=
ferent &quot;size&quot; (resource requirements) to starve each other</li><u=
l><li>side effect of needing to dynamically calculate storm &quot;slots&quo=
t; from mesos resource offers, my team has a framework change we are testin=
g that will improve that behavior.</li></ul><li>it&#39;s relatively complex=
 to operate since there are so many different moving parts / components</li=
><ul><li>mesos-master</li><li>mesos-slave/agent (it&#39;s being renamed)</l=
i><li>storm nimbus (MesosNimbus -&gt; the mesos scheduler)</li><li>storm su=
pervisor (MesosSupervisor -&gt; the mesos executor)</li><li>storm worker (t=
he mesos task)</li><li>ZooKeeper</li></ul><li>it probably won&#39;t work ni=
cely with all the fancy security stuff that has been added to Storm in 0.10=
.0+</li></ul><span class=3D"HOEnZb"><font color=3D"#888888"><div>- Erik</di=
v></font></span></div></div></div></div><div class=3D"HOEnZb"><div class=3D=
"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Feb =
1, 2016 at 12:14 PM, Bobby Evans <span dir=3D"ltr">&lt;<a href=3D"mailto:ev=
ans@yahoo-inc.com" target=3D"_blank">evans@yahoo-inc.com</a>&gt;</span> wro=
te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div><div style=3D"color:#000;backgro=
und-color:#fff;font-family:HelveticaNeue-Light,Helvetica Neue Light,Helveti=
ca Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:16px">We are cur=
rently adding in support for resource aware scheduling in a multi-tenant st=
and alone storm cluster.=C2=A0 It is still alpha quality but we plan on get=
ting it into production at Yahoo this quarter.=C2=A0 If you can wait that w=
ould be the preferred way I see to support your use case.<span><font color=
=3D"#888888"><br><div>=C2=A0</div><div>- Bobby
</div></font></span><div><div> <div><br><br></div><div style=3D"display:blo=
ck"> <div style=3D"font-family:HelveticaNeue-Light,Helvetica Neue Light,Hel=
vetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:16px"> <div =
style=3D"font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Gr=
ande,sans-serif;font-size:16px"> <div dir=3D"ltr"><font face=3D"Arial" size=
=3D"2"> On Monday, February 1, 2016 12:16 PM, Spico Florin &lt;<a href=3D"m=
ailto:spicoflorin@gmail.com" target=3D"_blank">spicoflorin@gmail.com</a>&gt=
; wrote:<br></font></div>  <br><br> <div><div>Hello!<br>I have an use case =
where we have to deploy many tpologies in a storm cluster. <br>1.we would l=
ike to know if running these topologies in combination with apache slider o=
ver yarn would bring us some benefits in terms of resource consumption? <br=
>2. in such cases (running many topolgies aprox 60) what are the best pract=
ices on how to run them over a cluster with a smart load balanced hardware =
resources consumption (cpu, ram)?<br>I look forward for your answers.<br> R=
egards,<br> Florin<br><br></div><br><br></div>  </div> </div>  </div></div>=
</div></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a11c3b1da21e75c052aef6c9d--