Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of ruj.sabya@gmail.com designates
 209.85.161.170 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=UgF9+bAmjYnLDXdG0+Zane9Zd37Cz8SvU3YxKZQ6bB02gcsm6e7Lk1PKbiPbgMvCRc
         o07dhx/gtiKlGjc/daxVs42Cx3vrb9os1IH41ksYj7hPpmhy9idsO2OFWwNJmepvXhRa
         dhTZFhxe6AFuBMrnZC/Gqn1D/28Mdk2p+dtkU=
MIME-Version: 1.0
From: Sabyasachi Ruj <ruj.sabya@gmail.com>
Date: Mon, 7 Mar 2011 16:11:55 +0530
Message-ID: <AANLkTimiBhwUyDSv_C3yLfLmbcWVzxd=3Qv6R-At_BmG@mail.gmail.com>
Subject: Task/Job distribution using ZooKeeper
To: user@zookeeper.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

I am planning to write an application which will have Worker processes
distributed across multiple=A0machines. One of them will be Leader which
will assign tasks to other processes. Designing the Leader elelection
process is quite simple: each process tries to create a ephemeral node
in the same path. Whoever is successful, becomes the leader. I got
this technique from Mahadev Konar's talk here:
http://developer.yahoo.com/blogs/ydn/posts/2009/08/hadoop_summit_zookeeper/
. But could not find any discussion about task/job distribution using
ZooKeeper.

I'll elaborate a little on the environment setup:

Suppose there are 10 worker maschines, each one runs a process, one of
them becomes the Leader. Tasks are submitted in the queue (may be
managed in MySQL), the Leader takes them and assigns to a worker. The
worker processes gets notified whenever a tasks is submitted by the
leader.

I think these jobs can be coordinated as child znodes for each worker node =
like:

/server/worker1/job1
/server/worker1/job2
/server/worker1/job3
/server/worker2/job1
/server/worker2/job2

To get an alert whenever a job is submitted, the workers can watch on
its corresponding znode. But again I've a doubt here. Is there a
chance in this case, that some jobs might get lost/delayed?

Step 1: Worker is watching on its zonde for jobs.
Step 2: Server submits a job X.
Step 3: Worker gets notified.
Step 4: Before setting the watch again, server submits another job Y.
Step 5: Now the worker sets the watch.

So, my questions are:

1. How to design the process of distributing the tasks evenly?
2. Was ZooKeeper designed for this use case?
3. In the example above, is there a chance that the worker may miss
notification for job Y?

--
Sabyasachi