hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HBASE-3106) a temporary solution of multi-thread compaction for 0.20.6
Date Wed, 16 Jul 2014 23:15:07 GMT

     [ https://issues.apache.org/jira/browse/HBASE-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell resolved HBASE-3106.
-----------------------------------

    Resolution: Not a Problem

> a temporary solution of multi-thread compaction for 0.20.6
> ----------------------------------------------------------
>
>                 Key: HBASE-3106
>                 URL: https://issues.apache.org/jira/browse/HBASE-3106
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>    Affects Versions: 0.20.6
>         Environment: 1master, 7 region servers, 4 * 7 clients(all clients run on region
server host), sequential put 
> configuration of each host is:
> memory: 16G (region server uses 10G)
> disk: 12T
> cpu: 8 core
>            Reporter: andychen
>         Attachments: multithread-compaction-0.20.6-v2.patch, multithread-compaction-0.20.6.patch
>
>
> Each region's memtable size is 64M, global memtable limit is 4G on one region server
> In heavy write scenario, especially disable WAL, sequential put  fills memtable to full
very quickly, and will generates too many storefiles in a short time.
> Then there are more and more compaction tasks put into compactionQueue, single thread
excutes compaction task too slowly, finally forms a vicious cycle.
> This will block some region's data importing for a long time
> We simplely using ThreadPoolExecutor as multi-thread compaction sulution
> (As we know, 0.90 or 0.92 will optimize memflush, compaction, open, close...etc using
multi-thread, but this magnificent version hasn't bee released yet)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message