hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <la...@apache.org>
Subject Re: secondary index feature
Date Sun, 22 Dec 2013 22:37:37 GMT
The devil is often in the details. On the surface it looks simple.

How specifically are the stale indexes ignored? Are the guaranteed to be no races?
Is deletion handled correctly?Does it work with multiple versions?
What happens when the client dies 1/2 way through an update?
It's easy to do eventually consistent indexes. Truly consistent indexes without transactions
are tricky.


Also, scanning an index and then doing point-gets against a main table is slow (unless the
index is very selective. The Phoenix team measured that there is only an advantage if the
index filters out 98-99% of the data).
So then one would revert to covered indexes and suddenly is not so easy 
to detect stale index entries.

I blogged about these issues here:
http://hadoop-hbase.blogspot.com/2012/10/musings-on-secondary-indexes.html
http://hadoop-hbase.blogspot.com/2012/10/secondary-indexes-part-ii.html

Phoenix has a (pretty involved) solution now that works around the fact that HBase has no
transactions.


-- Lars



________________________________
 From: Henning Blohm <henning.blohm@zfabrik.de>
To: user <user@hbase.apache.org> 
Sent: Sunday, December 22, 2013 2:11 AM
Subject: secondary index feature
 

Lately we have added a secondary index feature to a persistence tier 
over HBASE. Essentially we implemented what is described as "Dual-Write 
Secondary Index" in http://hbase.apache.org/book/secondary.indexes.html. 
I.e. while updating an entity, actually before writing the actual 
update, indexes are updated. Lookup via the index ignores stale entries. 
A recurring rebuild and clean out of stale entries takes care the 
indexes are trimmed and accurate.

None of this was terribly complex to implement. In fact, it seemed like 
something you could do generically, maybe not on the HBASE level itself, 
but as a toolbox / utility style library.

Is anybody on the list aware of anything useful already existing in that 
space?

Thanks,
Henning Blohm

*ZFabrik Software KG*

T:     +49 6227 3984255
F:     +49 6227 3984254
M:     +49 1781891820

Lammstrasse 2 69190 Walldorf

henning.blohm@zfabrik.de <mailto:henning.blohm@zfabrik.de>
Linkedin <http://www.linkedin.com/pub/henning-blohm/0/7b5/628>
ZFabrik <http://www.zfabrik.de>
Blog <http://www.z2-environment.net/blog>
Z2-Environment <http://www.z2-environment.eu>
Z2 Wiki <http://redmine.z2-environment.net>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message