hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Levin <magn...@gmail.com>
Subject hbase table as a queue.
Date Sat, 16 Jul 2011 16:38:25 GMT
Hello, we are thinking about using Hbase table as a simple queue which
will dispatch the work for a mapreduce job, as well as real time
fetching of data to present to end user.  In simple terms, suppose you
had a data source table and a queue table.  The queue table has a
smaller set of Rows that point to Values which in turn point to
Perma-set table, which has large collection of Rows.  (so Queue{Row,
Value} -> Perma-Set {Row, Value}).  Or Q-Value -> P-Row.   Our Goal is
to look up which Rows to retrieve from the Perma-Set table by looking
through the Queue.  Once the lookup into the Queue is done, the Row
from the Queue must be deleted to avoid the same process of Perma-Set
lookup be done twice; We expect many concurrent lookups to happen, so
I assume the first thing we need to do is to have a client that does
the work is acquire a lock on the Queue Row, process the work, then
Remove the Queue Row.

Has anyone done something similar before?  Any gotchas we should be away of?



View raw message