hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Koch <tho...@koch.ro>
Subject Re: Using Zookeeper to distribute tasks
Date Tue, 27 Apr 2010 08:14:07 GMT
David Rouchy:
> Hi all,
> 
> We are studying using ZooKeeper to manage configuration across multiple
> processes & server. What would be also interesting, as ZooKeeper know the
> list of process running, would be to use it to distribute tasks.
> 
> We have some long running tasks, so we used multiple servers to process
> multiple tasks in the same time. My idea would be to use multiple ephemeral
> watcher connected to one znode. A process will send data to this node when
>  a new task has to be launched. But here is my issue, I would like only one
>  watcher to be triggered of this change (a random watcher in the list). Is
>  there way to do such thing in ZooKeeper?
> 
> Regards,
> 
> David Rouchy

Hi David,

there's gearman[1], a client-server system to distribute tasks. I've mentioned 
it already two times on this list, so sorry for the repetition. Gearman 
servers and clients are availabe in different implementations that use the 
same protocol and are interchangeable AFAIK.
If somebody would build a gearman server using Zookeeper, there would be a 
whole ecosystem for ZK for free to conquer!

[1] http://gearman.org

Regards,

Thomas Koch, http://www.koch.ro

Mime
View raw message