hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reid Chan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
Date Wed, 15 Nov 2017 05:13:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Reid Chan updated HBASE-18309:
    Attachment: HBASE-18309.master.004.patch

* mv {{hbase.cleaner.threadpool.size}} to {{hbase.cleaner.scan.dir.concurrent.size}}
* add documentation for {{hbase.cleaner.scan.dir.concurrent.size}}
* keep sorting for (sub)dirs
* support multi threads in {{LogCleaner}} as well
* support online configuration change for both {{CleanerChore}} and {{LogCleaner}}
* add tests about online configuration change for both {{CleanerChore}} and {{LogCleaner}},
and tests passed locally

As concerned, it's possible for thread flood happen and drag NN down. And i have an idea (not
quite sure, and need some discussions), but would please allow me to file another jira for
that, it's too squeezed to put into one patch.

> Support multi threads in CleanerChore
> -------------------------------------
>                 Key: HBASE-18309
>                 URL: https://issues.apache.org/jira/browse/HBASE-18309
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: binlijin
>            Assignee: Reid Chan
>         Attachments: HBASE-18309.master.001.patch, HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
> There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.

This message was sent by Atlassian JIRA

View raw message