zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sabyasachi Ruj <ruj.sa...@gmail.com>
Subject Re: Task/Job distribution using ZooKeeper
Date Wed, 16 Mar 2011 14:05:17 GMT
Actually I need something similar:

Each consumer will have its own job queue. The leader will use round
robin to load balance. These jobs are never-ending and they run as
forked processes. A job will be signaled to stop by the consumer only
when the leader asks to do so.

I am planning implement this with the following znode hierarchy:

/nodes/nodeX/jobs/
	start/
		j1
		j2
		j3
	stop/
		j2

The consumer at nodeX wathches both 'start/' and 'stop/' znodes for
ZOO_CHILD_EVENT. If a node gets created under 'start' the consumer
forks a new worker. If a node gets created under 'stop', the consumer
signals the worker to stop.

Is this approach OK?

I am worried about if the consumer will always see events in 'start/'
and 'stop/' as they are applied by the leader.

For example, consider that leader executes this sequence:
	create /start/j2
	delete /stop/j2

In this situation, does the consumer have the gurantee that it always
see these events in that sequence? If the consumer sees the change
first at 'stop/' first and then at 'start/', this strategy won't work.

2011/3/8 Scott Fines <scottfines@gmail.com>
>
> Just to throw it in, https://github.com/openutility/menagerie also has
> blocking and synchronous queue implementations.
>
> If we had a deque, this would be a great opportunity for a work stealing
> scenario as well. You could give each consumer its own deque,  then the
> leader could use round robin to load balance. When a consumer has finished a
> task, it removes that item from the deque. Once a consumer runs out of items
> on its deque, it starts taking items off the back of someone else's. That
> way, when a machine dies or becomes slow, another machine can seamlessly
> take over its work.
>
> I'm working on a general purpose implementation for menagerie  now, but a
> deque shouldn't be that hard to make, if you follow the queue recipe
> mentioned above
> On Mar 8, 2011 2:50 AM, "Віталій Тимчишин" <tivv00@gmail.com> wrote:



--
Sabyasachi

Mime
View raw message