hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Brown <tombrow...@gmail.com>
Subject Re: Tracking down coprocessor pauses
Date Mon, 10 Sep 2012 20:39:47 GMT
Micheal,

We are using HBase to track the usage of our service. Specifically,
each client sends an update when they start a task, at regular
intervals during the task, and an update when they finish a task (and
then presumably they start another, continuing the cycle). Each user
has various attributes (which version of our software they're using,
their location, which task they're working on, etc), and we want to be
able to see stats in aggregate, and be able to drill-down into various
areas (similar to OLAP; Incidentally, we chose HBase because none of
the OLAP systems seemed to accept real-time updates).

The key design is a compound of:  [Attribute1 Attribute2... AttributeN].

Each row has roughly 10 cells, all of which represent counters; Some
require simple incrementing, but others require fancier bitwise
operations to properly increment (using HyperLogLog to estimate a
unique count).

The rows are stored with a 15-second granularity (everything from
0:00-0:15 is stored in one row, everything from 0:15-0:30 is in the
next, etc). The data is formatted such that you can get the
aggregation for a larger time period by combining all of the rows that
comprise that time frame. For the counter cells, this uses straight
addition. For the unique counters, bitwise operations are required.

The most frequently requested data has only one or two relevant
attributes. For example, we commonly want to see the stats of our
system broken out just by task. Of course, that makes writes a little
more difficult. When we have 1000's of users working on the same kind
of task, we'll receive a lot of concurrent updates to the row with
[attribute=TheTask]. HBase supports atomic increments, but not atomic
bitwise operations, so we were required to implement a locking
solution on our own.

There seemed to be a lot of problems with row-level locks, so we
decided to do the locking in the one place we could guarantee it: a
coprocessor. Within the coprocessor is logic to coalesce multiple
updates to the same row into a single HBase update. When performing
aggregations, a requested time period might summarize thousands of
rows into a single summary row. We thought that sending the entire set
over the network was overkill, especially since the aggregation
operations are fairly simple (addition and some bitwise calculations),
so the co-processor also contains code to perform aggregations.

I'm interested in improving the design, so any suggestions will be appreciated.

Thanks in advance,

--Tom

On Mon, Sep 10, 2012 at 12:45 PM, Michael Segel
<michael_segel@hotmail.com> wrote:
>
> On Sep 10, 2012, at 12:32 PM, Tom Brown <tombrown52@gmail.com> wrote:
>
>> We have our system setup such that all interaction is done through
>> co-processors. We update the database via a co-processor (it has the
>> appropriate logic for dealing with concurrent access to rows), and we
>> also query/aggregate via co-processor (since we don't want to send all
>> the data over the network).
>
> Could you expand on this? On the surface, this doesn't sound like a very good idea.
>

Mime
View raw message