hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Using Zookeeper to distribute tasks
Date Tue, 27 Apr 2010 07:34:32 GMT
The general way to do this is either

a) have lots of watchers who all try to create a single file when a watched
file changes.  This is very simple to code, but leads to a lot of
notifications when you have thousands of watchers.

b) arrange the watchers in a chain.  This is similar to the recommended
queue implementation.  See
http://hadoop.apache.org/zookeeper/docs/r3.2.1/recipes.html#sc_recipes_Queues
This requires careful implementation to deal correctly with session
expiration.

c) do as in (a), but spread the load by picking one of several files to
watch.  This still has herds jumping on a single file, but you can adjust
the number of files being used to keep this at a reasonable level.

On Tue, Apr 27, 2010 at 12:23 AM, David Rouchy <drouchy@gmail.com> wrote:

> My idea would be to use multiple ephemeral
> watcher connected to one znode. A process will send data to this node when
> a
> new task has to be launched. But here is my issue, I would like only one
> watcher to be triggered of this change (a random watcher in the list). Is
> there way to do such thing in ZooKeeper?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message