reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruv Mahajan <>
Subject Re: how to send message from task to driver
Date Wed, 08 Jun 2016 17:08:57 GMT
Apache Zookeeper is a good idea. If I am right, its sort of shared state
management kind of service? However, should we do it for REEF-1407 or later?

On Wed, Jun 8, 2016 at 10:04 AM, Markus Weimer <> wrote:

> On 2016-06-08 09:13, Dhruv Mahajan wrote:
>> in general I agree that if we can introduce an extra push based
>> handler for task for this that would be good.
> The challenge with that is the potential to overwhelm the Driver. If we
> have 1000s of Tasks all doing that at the same time, the Driver will be the
> bottleneck. This isn't an issue for a few dozens of Tasks, though.
> For very large deployments, we might want to look into Apache ZooKeeper to
> derive consensus amongst the Tasks.
> Markus

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message