From user-return-2846-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org Mon Mar 07 16:44:04 2011 Return-Path: Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: (qmail 66009 invoked from network); 7 Mar 2011 16:44:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Mar 2011 16:44:04 -0000 Received: (qmail 45610 invoked by uid 500); 7 Mar 2011 16:44:03 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 45545 invoked by uid 500); 7 Mar 2011 16:44:03 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 45536 invoked by uid 99); 7 Mar 2011 16:44:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 16:44:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com designates 209.85.212.42 as permitted sender) Received: from [209.85.212.42] (HELO mail-vw0-f42.google.com) (209.85.212.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 16:43:55 +0000 Received: by vws10 with SMTP id 10so5114066vws.15 for ; Mon, 07 Mar 2011 08:43:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=pUPnX5Qcd4M8dftA2b05EPqM5d/FVGfZTbLl2hkqbrk=; b=X7Nd1XpG5amnqqKdVCy123AK85Ew8oXhyam5TB7/Qm1j13og/j9EFurIgdbUzsJ0lP 0AS4VfcksPqpLerZR2deYBNHgAqjzDhCEKTqZDCn2C8nX+82hUByKJg3WRUQoGmVTylt Q7KeOMiZZV5VHX6AVnbDYMS57bUQaYT6aaYHE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=ja5K0Q2KISg/aIrRq+atUU3Ld+WyFQHJVGEEFSwSz2ll03fTP/TQqZEQkvYszmI5MJ uOr1JRcYCvspiODw0ZZGJ3Auwd0PQnUH5OoHwyaszWPSWKULw/AU+dU5ZFZ5VGmOcxm0 05/h7ShfYNPEcihFkbebag+mLPCtte2Fqcn/w= Received: by 10.52.98.106 with SMTP id eh10mr5692230vdb.109.1299516214136; Mon, 07 Mar 2011 08:43:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.167.37 with HTTP; Mon, 7 Mar 2011 08:43:14 -0800 (PST) In-Reply-To: References: From: Ted Dunning Date: Mon, 7 Mar 2011 08:43:14 -0800 Message-ID: Subject: Re: Task/Job distribution using ZooKeeper To: user@zookeeper.apache.org Cc: Sabyasachi Ruj Content-Type: multipart/alternative; boundary=20cf307f319252d0a6049de7321d X-Virus-Checked: Checked by ClamAV on apache.org --20cf307f319252d0a6049de7321d Content-Type: text/plain; charset=UTF-8 Yes and no. The worker may miss a notification. But that isn't a problem because it shouldn't miss any data. The idea is that only one notification will go to the worker. If the server changes the file again before the worker gets around to looking at the data, no additional notification will be given to the worker. The worker will set a new notification in the same operation as it reads the data so it will get both updates and a new notification will be set to tell it about any further updates. I think you design is a bit short on notifying the master of work that is finished. Also, I don't think you handle worker failure in a nice resilient way. Camille's suggestion of a single job queue can be extended to handle both issues. Good luck! On Mon, Mar 7, 2011 at 2:41 AM, Sabyasachi Ruj wrote: > 3. In the example above, is there a chance that the worker may miss > notification for job Y? > --20cf307f319252d0a6049de7321d--