storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Weathers <eweath...@groupon.com>
Subject Re: running multiple topologies in same cluster
Date Mon, 01 Feb 2016 23:00:17 GMT
hi Spico,

As Bobby said, native Storm is going to have better support for this soon.

FWIW, there is also the storm-on-mesos project, which we've been running on
for almost 2 years at Groupon.

   - https://github.com/mesos/storm

Caveats:

   - storm's logviewer is unsupported
   - scheduling can be suboptimal, causing topologies of different "size"
   (resource requirements) to starve each other
      - side effect of needing to dynamically calculate storm "slots" from
      mesos resource offers, my team has a framework change we are testing that
      will improve that behavior.
   - it's relatively complex to operate since there are so many different
   moving parts / components
      - mesos-master
      - mesos-slave/agent (it's being renamed)
      - storm nimbus (MesosNimbus -> the mesos scheduler)
      - storm supervisor (MesosSupervisor -> the mesos executor)
      - storm worker (the mesos task)
      - ZooKeeper
   - it probably won't work nicely with all the fancy security stuff that
   has been added to Storm in 0.10.0+

- Erik

On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans <evans@yahoo-inc.com> wrote:

> We are currently adding in support for resource aware scheduling in a
> multi-tenant stand alone storm cluster.  It is still alpha quality but we
> plan on getting it into production at Yahoo this quarter.  If you can wait
> that would be the preferred way I see to support your use case.
>
> - Bobby
>
>
> On Monday, February 1, 2016 12:16 PM, Spico Florin <spicoflorin@gmail.com>
> wrote:
>
>
> Hello!
> I have an use case where we have to deploy many tpologies in a storm
> cluster.
> 1.we would like to know if running these topologies in combination with
> apache slider over yarn would bring us some benefits in terms of resource
> consumption?
> 2. in such cases (running many topolgies aprox 60) what are the best
> practices on how to run them over a cluster with a smart load balanced
> hardware resources consumption (cpu, ram)?
> I look forward for your answers.
> Regards,
> Florin
>
>
>
>

Mime
View raw message