hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: use hbase to store realtime logs
Date Wed, 30 Sep 2009 03:46:42 GMT
You could use hbase to do this.  Why not just put them into hdfs (Check out
tech like facebook's scribe).   If you do put them to hbase, make sure you
provision your cluster with sufficient firepower (measure write rate to a
single node then size appropriately giving yourself a decent amount of elbow
room to grow in).

Unix timestamp is not enough to uniquely specify log entries, not if you are
doing 100k a second.  You may have to design a better key than this.  Add a
sequence number or some such.


Other architectures that you might consider are writing files locally and
then on a period pushing to hdfs.

On Tue, Sep 29, 2009 at 6:17 PM, Zheng Shao <zshao@facebook.com> wrote:

> Is it a good use case to store realtime logs into hbase?
> I am thinking of using unix timestamp as the key, and we have 100K/rows per
> seconds, and 100 bytes per row (about 10MB/second).
> Users can do range query to get the latest rows. Periodically, we rotate
> the tables.
> In my case, the key is monotonically increasing but HBase is general enough
> to take random keys.
> I am not sure this is a good use case for HBase.
> Does anybody have similar use case? Does HBase work well for this?
> Zheng

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message