hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Using Zookeeper to distribute tasks
Date Wed, 19 May 2010 15:34:01 GMT
That's the beauty of zk, it's provides primitives that allow you much 
flexibility in what/how you implement something. Perhaps you could 
sketch out a bit more detail on what you had in mind, requirements/goals 
and such? What I mean is, from a high level coordination perspective, 
then we could see how that might map down to zk operations.

Patrick

On 05/19/2010 03:29 AM, David Rouchy wrote:
> Hi all,
>
> I'm studying ZooKeeper to see how we can use it in a project. I'm still a
> little big puzzled by the usage of zookeeper for distributing tasks. The
> usage of queue does not fully fit the bill. I was just wondering if it would
> be interesting / realistic to use a kind of "election" to decide on which
> znode you want to send the task?
>
> David Rouchy
>
> On Wed, Apr 28, 2010 at 7:39 AM, David Rouchy<drouchy@gmail.com>  wrote:
>
>> Thank you for your quick answers. I'll have a look.
>>
>> David
>>
>>
>> On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch<thomas@koch.ro>  wrote:
>>
>>> David Rouchy:
>>>> Hi all,
>>>>
>>>> We are studying using ZooKeeper to manage configuration across multiple
>>>> processes&  server. What would be also interesting, as ZooKeeper know
>>> the
>>>> list of process running, would be to use it to distribute tasks.
>>>>
>>>> We have some long running tasks, so we used multiple servers to process
>>>> multiple tasks in the same time. My idea would be to use multiple
>>> ephemeral
>>>> watcher connected to one znode. A process will send data to this node
>>> when
>>>>   a new task has to be launched. But here is my issue, I would like only
>>> one
>>>>   watcher to be triggered of this change (a random watcher in the list).
>>> Is
>>>>   there way to do such thing in ZooKeeper?
>>>>
>>>> Regards,
>>>>
>>>> David Rouchy
>>>
>>> Hi David,
>>>
>>> there's gearman[1], a client-server system to distribute tasks. I've
>>> mentioned
>>> it already two times on this list, so sorry for the repetition. Gearman
>>> servers and clients are availabe in different implementations that use the
>>> same protocol and are interchangeable AFAIK.
>>> If somebody would build a gearman server using Zookeeper, there would be a
>>> whole ecosystem for ZK for free to conquer!
>>>
>>> [1] http://gearman.org
>>>
>>> Regards,
>>>
>>> Thomas Koch, http://www.koch.ro
>>>
>>
>>
>

Mime
View raw message