hi Spico,
As Bobby said, native Storm is going to have better support for this soon.
FWIW, there is also the storm-on-mesos project, which we've been running on
for almost 2 years at Groupon.
- https://github.com/mesos/storm
Caveats:
- storm's logviewer is unsupported
- scheduling can be suboptimal, causing topologies of different "size"
(resource requirements) to starve each other
- side effect of needing to dynamically calculate storm "slots" from
mesos resource offers, my team has a framework change we are testing that
will improve that behavior.
- it's relatively complex to operate since there are so many different
moving parts / components
- mesos-master
- mesos-slave/agent (it's being renamed)
- storm nimbus (MesosNimbus -> the mesos scheduler)
- storm supervisor (MesosSupervisor -> the mesos executor)
- storm worker (the mesos task)
- ZooKeeper
- it probably won't work nicely with all the fancy security stuff that
has been added to Storm in 0.10.0+
- Erik
On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans <evans@yahoo-inc.com> wrote:
> We are currently adding in support for resource aware scheduling in a
> multi-tenant stand alone storm cluster. It is still alpha quality but we
> plan on getting it into production at Yahoo this quarter. If you can wait
> that would be the preferred way I see to support your use case.
>
> - Bobby
>
>
> On Monday, February 1, 2016 12:16 PM, Spico Florin <spicoflorin@gmail.com>
> wrote:
>
>
> Hello!
> I have an use case where we have to deploy many tpologies in a storm
> cluster.
> 1.we would like to know if running these topologies in combination with
> apache slider over yarn would bring us some benefits in terms of resource
> consumption?
> 2. in such cases (running many topolgies aprox 60) what are the best
> practices on how to run them over a cluster with a smart load balanced
> hardware resources consumption (cpu, ram)?
> I look forward for your answers.
> Regards,
> Florin
>
>
>
>
|