hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
Date Tue, 11 Dec 2012 18:57:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529218#comment-13529218

Sergey Shelukhin commented on HBASE-6466:

bq. Yeah, we don't make use of these flush markers.
Created HBASE-7329 to remove.

bq.    ... with lock held between start and complete entries. If this lock is kept exclusive,
it will cause flush threads to serialize on it.
bq. Ok. Were you able to make this happen Sergey?
I took the patch with locks already fixed. I can try but it seems there's already consensus
on this :)
bq. On jds' concern, its this one: 'Also if this patch doesn't modify the behavior of HLog.startCacheFlush
and HLog.completeCacheFlush WRT the cacheFlushLock I can't see how it could make things any
bq. So, your reentrant lock is how you address his concern?
Yeah, I think this locking was meant by the comment.
bq. Any guards against us flushing same memstore concurrently: i.e. we are already flushing
it and we start in flushing it again in a concurrent thread?
There's code in flushcache that prevents this via writestate:
      synchronized (writestate) {
        if (!writestate.flushing && writestate.writesEnabled) {
          this.writestate.flushing = true;
        } else {
close calls internal flush directly but it also sets writesEnabled to false.
> Enable multi-thread for memstore flush
> --------------------------------------
>                 Key: HBASE-6466
>                 URL: https://issues.apache.org/jira/browse/HBASE-6466
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>         Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.1.patch, HBASE-6466v3.patch,
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is
often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment : 
> random writting;upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per
regionserver;row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per
regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver,
appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver,
Flush:18.6MB/s per regionserver

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message