hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reid Chan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
Date Thu, 23 Nov 2017 08:07:01 GMT

     [ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Reid Chan updated HBASE-18309:
------------------------------
    Release Note: 
After HBASE-18309,
1. we could use multiple threads to scan archive directories (including data and oldWALs)
through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning
the concrete size, but no more than available cpu cores) and double (between 0 and 1, meaning
ratio of available cpu cores) value and defaults to 0.5.
2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size,
2 by default.
3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size
support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and
hbase.regionserver.hfilecleaner.small.thread.count does.
4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count,
hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into
account when setting this config to avoid thread flooding.

  was:
After HBASE-18309 we could use multiple threads to scan archive directories (including data
and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer
(meaning the concrete size, but no more than available cpu cores) and double (between 0 and
1, meaning ratio of available cpu cores) value and defaults to 0.5.
We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size,
2 by default.
Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count,
hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into
account when setting this config to avoid thread flooding.
In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size
support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and
hbase.regionserver.hfilecleaner.small.thread.count does.


> Support multi threads in CleanerChore
> -------------------------------------
>
>                 Key: HBASE-18309
>                 URL: https://issues.apache.org/jira/browse/HBASE-18309
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: binlijin
>            Assignee: Reid Chan
>             Fix For: 3.0.0, 2.0.0-beta-1
>
>         Attachments: HBASE-18309.master.001.patch, HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, HBASE-18309.master.007.patch,
HBASE-18309.master.008.patch, HBASE-18309.master.009.patch, HBASE-18309.master.010.patch,
HBASE-18309.master.011.patch, HBASE-18309.master.012.patch, space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.
> What's more, currently there's only one thread iterating the archive directory, and we
could use multiple threads cleaning sub directories in parallel to speed it up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message