Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: Mahadev Konar <mahadev@yahoo-inc.com>
Subject: Re: feed queue fetcher with hadoop/zookeeper/gearman?
Date: Mon, 12 Apr 2010 19:49:01 +0200
User-Agent: KMail/1.12.4 (Linux/2.6.32-4-amd64; KDE/4.3.4; x86_64; ; )
Cc: zookeeper-user@hadoop.apache.org,
 common-user@hadoop.apache.org,
 gearman@googlegroups.com
References: <C7E89C75.33584%mahadev@yahoo-inc.com>
In-Reply-To: <C7E89C75.33584%mahadev@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201004121949.01427.thomas@koch.ro>

Mahadev Konar:
> Hi Thomas,
>   There are a couple of projects inside Yahoo! that use ZooKeeper as an
> event manager for feed processing.
> 
> I am little bit unclear on your example below. As I understand it-
> 
> 1. There are 1 million feeds that will be stored in Hbase.
> 2. A map reduce job will be run on these feeds to find out which feeds need
> to be fetched.
> 3. This will create queues in ZooKeeper to fetch the feeds
> 4.  Workers will pull items from this queue and process feeds
> 
> Did I understand it correctly? Also, if above is the case, how many queue
> items would you anticipate be accumulated every hour?
Yes. That's exactly what I'm thinking about. Currently one node processes like 
20000 Feeds an hour and we have 5 feed-fetch-nodes. This would mean ~100000 
queue items/hour. Each queue item should carry some meta informations, most 
important the feed items, that are already known to the system so that only 
new items get processed.

Thomas Koch, http://www.koch.ro