hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6980) Parallel Flushing Of Memstores
Date Tue, 16 Oct 2012 05:35:03 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476756#comment-13476756

ramkrishna.s.vasudevan commented on HBASE-6980:

I can try to explain what i see from the code.  I am sure you would have checked but just
in case you have not.  I may be wrong so pls correct me.
>From the flush code we see that once we do the startFlush() we acquire the cacheFlushLock.
The same is cleared in completeCacheFlush().  It is here where we write the latest seqid corresponding
to that flush.  Suppose the seq ids 2,3,4 are getting flushed we write a seq id 5 for the
current flush entry.  
As the javadoc too says, now suppose a log rolling happens without cacheflush lock
synchronized (updateLock) {
        // Clean up current writer.
        Path oldFile = cleanupCurrentWriter(currentFilenum);
        this.writer = nextWriter;
        this.hdfs_out = nextHdfsOut;
The log writer may be changed and the completeCacheflush may write to a new file (if am not
Also currently whenever we do flush the oldest seq id for the region is removed from the lastSeqWritten
and the same is again populated back with a sort of encoded name for the region
Long oldseq =
        lastSeqWritten.put(getSnapshotName(encodedRegionName), seq);
This was done for a dataloss issue by FB.  So may be if we don acquire the cacheFlushLock
on rollWriter, the rollWriter() may see some regions(with encoded name) which has min seq
id and will try to flush them too in
 private byte [] getOldestRegion(final Long oldestOutstandingSeqNum) {
    byte [] oldestRegion = null;
    for (Map.Entry<byte [], Long> e: this.lastSeqWritten.entrySet()) {
      if (e.getValue().longValue() == oldestOutstandingSeqNum.longValue()) {
        // Key is encoded region name.
        oldestRegion = e.getKey();
    return oldestRegion;
May be other experts can give a better answer if am not right here. Thanks.
> Parallel Flushing Of Memstores
> ------------------------------
>                 Key: HBASE-6980
>                 URL: https://issues.apache.org/jira/browse/HBASE-6980
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
> For write dominated workloads, single threaded memstore flushing is an unnecessary bottleneck.
With a single flusher thread, we are basically not setup to take advantage of the aggregate
throughput that multi-disk nodes provide.
> * For puts with WAL enabled, the bottleneck is more likely the "single" WAL per region
server. So this particular fix may not buy as much unless we unlock that bottleneck with multiple
commit logs per region server. (Topic for a separate JIRA-- HBASE-6981).
> * But for puts with WAL disabled (e.g., when using HBASE-5783 style fast bulk imports),
we should be able to support much better ingest rates with parallel flushing of memstores.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message