storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Spico Florin <spicoflo...@gmail.com>
Subject Re: running multiple topologies in same cluster
Date Thu, 04 Feb 2016 10:44:43 GMT
Hello!
Thank you all for your answers! I guess I'll wait for adding in support for
resource aware scheduling in a multi-tenant stand alone storm cluster.
Regards,
 Florin

On Tue, Feb 2, 2016 at 1:00 AM, Erik Weathers <eweathers@groupon.com> wrote:

> hi Spico,
>
> As Bobby said, native Storm is going to have better support for this soon.
>
> FWIW, there is also the storm-on-mesos project, which we've been running
> on for almost 2 years at Groupon.
>
>    - https://github.com/mesos/storm
>
> Caveats:
>
>    - storm's logviewer is unsupported
>    - scheduling can be suboptimal, causing topologies of different "size"
>    (resource requirements) to starve each other
>       - side effect of needing to dynamically calculate storm "slots"
>       from mesos resource offers, my team has a framework change we are testing
>       that will improve that behavior.
>    - it's relatively complex to operate since there are so many different
>    moving parts / components
>       - mesos-master
>       - mesos-slave/agent (it's being renamed)
>       - storm nimbus (MesosNimbus -> the mesos scheduler)
>       - storm supervisor (MesosSupervisor -> the mesos executor)
>       - storm worker (the mesos task)
>       - ZooKeeper
>    - it probably won't work nicely with all the fancy security stuff that
>    has been added to Storm in 0.10.0+
>
> - Erik
>
> On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans <evans@yahoo-inc.com> wrote:
>
>> We are currently adding in support for resource aware scheduling in a
>> multi-tenant stand alone storm cluster.  It is still alpha quality but we
>> plan on getting it into production at Yahoo this quarter.  If you can wait
>> that would be the preferred way I see to support your use case.
>>
>> - Bobby
>>
>>
>> On Monday, February 1, 2016 12:16 PM, Spico Florin <spicoflorin@gmail.com>
>> wrote:
>>
>>
>> Hello!
>> I have an use case where we have to deploy many tpologies in a storm
>> cluster.
>> 1.we would like to know if running these topologies in combination with
>> apache slider over yarn would bring us some benefits in terms of resource
>> consumption?
>> 2. in such cases (running many topolgies aprox 60) what are the best
>> practices on how to run them over a cluster with a smart load balanced
>> hardware resources consumption (cpu, ram)?
>> I look forward for your answers.
>> Regards,
>> Florin
>>
>>
>>
>>
>

Mime
View raw message