hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu Shaohui (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
Date Thu, 17 Apr 2014 10:17:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972817#comment-13972817
] 

Liu Shaohui commented on HBASE-6466:
------------------------------------

[~zjushch]
Very curious about this patch. 

In current codebase,  the write of hlog is many small pieces write and sync, and the flush
of memstore are batch write. So i think the throughput of one -thread flush  is larger than
the throughput of write of hlog. That means one flush thread is enough for most cases. 

What's scenario for multi-thread memstore flush? High-throughput write without wal? 



> Enable multi-thread for memstore flush
> --------------------------------------
>
>                 Key: HBASE-6466
>                 URL: https://issues.apache.org/jira/browse/HBASE-6466
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>    Affects Versions: 0.95.2
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Critical
>             Fix For: 0.95.0
>
>         Attachments: 6466-v6.patch, 6466-v7.patch, HBASE-6466-v4.patch, HBASE-6466-v4.patch,
HBASE-6466-v5.patch, HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.1.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is
often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment : 
> random writting;upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per
regionserver;row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per
regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver,
appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver,
Flush:18.6MB/s per regionserver



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message