Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <25002264.post@talk.nabble.com>
Date: Mon, 17 Aug 2009 00:26:45 -0700 (PDT)
From: Peter Rietzler <peter.rietzler@smarter-ecommerce.com>
To: hbase-user@hadoop.apache.org
Subject: Re: Feed Aggregator Schema
In-Reply-To: <e8666cd10908140855g63900fc7vbcec5de2178e51c0@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <e8666cd10908140855g63900fc7vbcec5de2178e51c0@mail.gmail.com>


Hi 

In our project we are handling event lists where we have similar
requirements. We do ordering by choosing our row keys wisely. We use the
following key for our events (they should be ordered by time in ascending
order):

eventListName/yyyyMMddHHmmssSSS-000[-111]

where eventListName is the name of the event list and 000 is a three digit
instance id to disambiguate between different running instances of
application, and -111 is optional to disambiguate events that occured in the
same millisecond on one instance. 

We additionally insert and artifical row for each day with the id

eventListName/yyyyMMddHHmmssSSS

This allows us to start scanning at the beginning of each day without
searching through the event list.

You need to be aware of the fact that if you have a very high load of
inserts, then always one hbase region server is busy inserting while the
others are idle ... if that's a problem for you, you have to find different
keys for your purpose. 

You could also use an HBase index table but I have no experience with it and
I remember an email on the mailing list that this would double all requests
because the API would first lookup the index table and then the original
table ??? (please correct me if this is not right ...)

Kind regards, 
Peter


Andrei Savu wrote:
> 
> Hello,
> 
> I am working on a project involving monitoring a large number of
> rss/atom feeds. I want to use hbase for data storage and I have some
> problems designing the schema. For the first iteration I want to be
> able to generate an aggregated feed (last 100 posts from all feeds in
> reverse chronological order).
> 
> Currently I am using two tables:
> 
> Feeds: column families Content and Meta : raw feed stored in Content:raw
> Urls: column families Content and Meta : raw post version stored in
> Content:raw and the rest of the data found in RSS stored in Meta
> 
> I need some sort of index table for the aggregated feed. How should I
> build that? Is hbase a good choice for this kind of application?
> 
> In other words: Is it possible( in hbase) to design a schema that
> could efficiently answer to queries like the one listed bellow?
> 
> SELECT data FROM Urls ORDER BY date DESC LIMIT 100
> 
> Thanks.
> 
> --
> Savu Andrei
> 
> Website: http://www.andreisavu.ro/
> 
> 

-- 
View this message in context: http://www.nabble.com/Feed-Aggregator-Schema-tp24974071p25002264.html
Sent from the HBase User mailing list archive at Nabble.com.