reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruv Mahajan <dhruv.maha...@gmail.com>
Subject Re: how to send message from task to driver
Date Wed, 08 Jun 2016 17:08:57 GMT
Apache Zookeeper is a good idea. If I am right, its sort of shared state
management kind of service? However, should we do it for REEF-1407 or later?

On Wed, Jun 8, 2016 at 10:04 AM, Markus Weimer <markus@weimo.de> wrote:

> On 2016-06-08 09:13, Dhruv Mahajan wrote:
>
>> in general I agree that if we can introduce an extra push based
>> handler for task for this that would be good.
>>
>
> The challenge with that is the potential to overwhelm the Driver. If we
> have 1000s of Tasks all doing that at the same time, the Driver will be the
> bottleneck. This isn't an issue for a few dozens of Tasks, though.
>
> For very large deployments, we might want to look into Apache ZooKeeper to
> derive consensus amongst the Tasks.
>
> Markus
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message