hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Baranau <alex.barano...@gmail.com>
Subject Understanding compacting memstore/HLog before flush
Date Wed, 02 May 2012 04:00:30 GMT

Could you please tell me if I correctly understand this problem...

Example behavior 1:
* create table
* do 10 operations: insert cell, override (given that versions # configured
to 1) it, override, ... override.
* after flushing memstore with these edits, all of them getting written to

Ideally, in this situation one edit should be performed (resulting value of
cell). I.e. only "current visible state" of memstore should be flushed as
opposed to flushing all the edits from HLog. This will have a lot of
benefits (e.g. reducing data amount to flush -> may be less frequent
flushing needing -> less freq compactions, etc. operations), esp in
particular use-cases (like using counters, or updating some "aggregated

The problem, as I understand (correct me here, please if I'm wrong) is that
it is not an easy thing to do, mainly because
1) additional resource management burden (flushing large memstore isn't
2) compaction may add a lot of unnecessary overhead (so that in some cases
there will be no actual benefit from it), may make flushing much slower,
which can bring a lot of issues
3) edits flushed from memstore and HLog edits should be kept in sync,
because we want the flush process to be reliable. I.e. if it fails in the
middle we should be able to restore the state from HLog. Keeping memstore
and HLog in sync during compaction (and we would need partial compaction of
some older data of the memstore) is difficult.
4) anything else?

Esp. 3rd point - am I getting it right?

Alex Baranau

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message