hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robertis Tongbram <rtongb...@gmail.com>
Subject A different sort of BitComparator
Date Mon, 24 Sep 2012 18:40:48 GMT
Hi all,

We have this scenario, where we pre-calculate a condition ( based on 7/8
dimensions - like  age, gender, occupation). We then use this calculated
value as column qualifier ( not as Rowkeys).
To optimize storage, we use a bit set to represent the dimensions ( for
example , in gender, we use 0 to represent female and 1 to represent male,
so we use only 1 bit to represent gender).  When we insert data to Hbase,
we do this calculation ( using BigInteger since we use Java 6) . And during
lookup, we do the similar bit-wise calculation and come up with the right
column qualifier. Calculation is a simple shift+OR operation. We also had
to re-implement compareTo() method of BitComparator class to mask and
compare ( for example, if we want to look up occupation, we mask the other
conditions and compare with the value from KeyValue). With this , the
response times almost reduced to half from the previous filter of using
RegEx Filter ( Earlier, we used Regex to compare the column qualifier where
the conditions are concatenated using a delimiter) .

I wanted to ask the Hbase experts here, if this is the right way of doing ,
or is there a better more scalable and maintainable way.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message