zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Fines <scottfi...@gmail.com>
Subject Re: Task/Job distribution using ZooKeeper
Date Tue, 08 Mar 2011 12:52:28 GMT
Just to throw it in, https://github.com/openutility/menagerie also has
blocking and synchronous queue implementations.

If we had a deque, this would be a great opportunity for a work stealing
scenario as well. You could give each consumer its own deque,  then the
leader could use round robin to load balance. When a consumer has finished a
task, it removes that item from the deque. Once a consumer runs out of items
on its deque, it starts taking items off the back of someone else's. That
way, when a machine dies or becomes slow, another machine can seamlessly
take over its work.

I'm working on a general purpose implementation for menagerie  now, but a
deque shouldn't be that hard to make, if you follow the queue recipe
mentioned above
On Mar 8, 2011 2:50 AM, "Віталій Тимчишин" <tivv00@gmail.com> wrote:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message