hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-5476) Merge flush/merging compaction; save on i/o by merging an existing file with content memstore
Date Sat, 25 Feb 2012 01:15:48 GMT
Merge flush/merging compaction; save on i/o by merging an existing file with content memstore
---------------------------------------------------------------------------------------------

                 Key: HBASE-5476
                 URL: https://issues.apache.org/jira/browse/HBASE-5476
             Project: HBase
          Issue Type: Improvement
            Reporter: stack


Compactions are slow.  To improve, we can work on  making compactions faster (TODO) or make
it so compactions have less work to do.  This issue is about the later; doing something the
bigtable paper talks off where the content of memory is merged with existing content in the
filesystem (I'm sure we've discussed this a bunch in the past but can't find an explicit issue
for it).

We save on the reading of the size of a memstore from the filesystem if we merge flush/merge
compact.

BT seems to include memstore content in minor compactions.  We should look at doing that (would
snapshot at start of compaction and then would integrate the snapshot into resultant compacted
file).  Alternatively we could flush into a small file that is already in the filesystem (this
could be tougher given that we have this flush-compaction separation at the moment... how
would we make it so the file we're merging into is not picked up for a normal compaction).

Doing the merge compaction/flush merge would slow down flushes.  It could back up memory into
the global barrier such that we stop taking on writes altogether.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message