Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 74.125.92.26 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=XuXcyAm2qWdRK3j3QAKeO4UEbBdQfTqbehj9rq/9C5pH63bQ3kRz2tKedgpCvWa0NW
         ow3VbSt0lZO4D7cW6UMDZgyttHKAhQaEfSdChbFTApwEs3AMcS+qdrwkVCYNz1ZuoR+c
         toEcj5gYK6IoXwIMtl2BeK3Cy0GTDS+ulneSU=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: 
 <CD2E0A273BDAAD409C70BB3F48EE6A126CC30A7A97@SC-MBXC1.TheFacebook.com>
References: 
 <CD2E0A273BDAAD409C70BB3F48EE6A126CC30A7A97@SC-MBXC1.TheFacebook.com>
Date: Tue, 29 Sep 2009 20:46:42 -0700
Message-ID: <7c962aed0909292046n6231b9a5xd5aded9155f0903@mail.gmail.com>
Subject: Re: use hbase to store realtime logs
From: stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09fa216fe0db38d0474c3611e

--00c09fa216fe0db38d0474c3611e
Content-Type: text/plain; charset=ISO-8859-1

You could use hbase to do this.  Why not just put them into hdfs (Check out
tech like facebook's scribe).   If you do put them to hbase, make sure you
provision your cluster with sufficient firepower (measure write rate to a
single node then size appropriately giving yourself a decent amount of elbow
room to grow in).

Unix timestamp is not enough to uniquely specify log entries, not if you are
doing 100k a second.  You may have to design a better key than this.  Add a
sequence number or some such.

St.Ack

Other architectures that you might consider are writing files locally and
then on a period pushing to hdfs.

On Tue, Sep 29, 2009 at 6:17 PM, Zheng Shao <zshao@facebook.com> wrote:

> Is it a good use case to store realtime logs into hbase?
>
> I am thinking of using unix timestamp as the key, and we have 100K/rows per
> seconds, and 100 bytes per row (about 10MB/second).
> Users can do range query to get the latest rows. Periodically, we rotate
> the tables.
>
> In my case, the key is monotonically increasing but HBase is general enough
> to take random keys.
> I am not sure this is a good use case for HBase.
>
> Does anybody have similar use case? Does HBase work well for this?
>
> Zheng
>
>

--00c09fa216fe0db38d0474c3611e--