hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
Date Tue, 11 Dec 2012 06:39:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528726#comment-13528726

stack commented on HBASE-6466:

bq. I guess one could achieve the same goal with threadpool and counter to limit the concurrency
(except that these threads will be easier to starve) but it seems like a roundabout way to
do is.

Thanks for taking a look.  The diff between thread pool and what is running here makes sense.

bq. At the cost of few pointers (it currently refers to parent fields

Thats ok.  Was just asking.  Its fine the way it is especially later where you point out that
you are making use of Server from the parent class.

bq. The behavior should be the same - the uncaught exception handler is set on new runnable-s
the same way as on the old one.

Ok.  Was worried the exception would be dropped on the floor.  Sounds good.

bq. ....WAL has entries for cache flush (which according to discussion in the linked FB JIRA
might be unnecessary)...

Yeah, we don't make use of these flush markers.

bq. What is going on w/ blockSignal?

I ask because I'm wary when new locks added.  This is an addition in a little exercised piece
of the code.  Was looking for justification on why the addition.  On the face of it, it seems
plain enough.

bq. ... with lock held between start and complete entries. If this lock is kept exclusive,
it will cause flush threads to serialize on it.

Ok.  Were you able to make this happen Sergey?

On jds' concern, its this one: 'Also if this patch doesn't modify the behavior of HLog.startCacheFlush
and HLog.completeCacheFlush WRT the cacheFlushLock I can't see how it could make things any

So, your reentrant lock is how you address his concern?

Any guards against us flushing same memstore concurrently: i.e. we are already flushing it
and we start in flushing it again in a concurrent thread?

> Enable multi-thread for memstore flush
> --------------------------------------
>                 Key: HBASE-6466
>                 URL: https://issues.apache.org/jira/browse/HBASE-6466
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>         Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.1.patch, HBASE-6466v3.patch,
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is
often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment : 
> random writting;upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per
regionserver;row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per
regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver,
appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver,
Flush:18.6MB/s per regionserver

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message