hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
Date Thu, 23 Nov 2017 03:18:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yu Li updated HBASE-18309:
--------------------------
     Hadoop Flags: Reviewed
     Release Note: After HBASE-18309 we could use multiple threads to scan archive directories
(including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which
supports both integer (meaning the concrete size) and double (between 0 and 1, meaning ratio
of available cpu cores) value and defaults to 0.5. Please take hbase.regionserver.hfilecleaner.large.thread.count
and hbase.regionserver.hfilecleaner.small.thread.count into account when setting this config
to avoid thread flooding. We also support using multiple threads to clean wals in a single
directory through hbase.oldwals.cleaner.thread.size, 2 by default.
    Fix Version/s: 2.0.0-beta-1
                   3.0.0
      Description: 
There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find this
is not enough. The number of files under oldWALs reach the max-directory-items limit of HDFS
and cause region server crash, so we use multi threads for LogCleaner and the crash not happened
any more.

What's more, currently there's only one thread iterating the archive directory, and we could
use multiple threads cleaning sub directories in parallel to speed it up.

  was:There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.

      Component/s:     (was: wal)

[~reidchan] please check the release note and feel free to refine it. It's recommended to
add release note when introducing new properties, so people could better know how to use them.

> Support multi threads in CleanerChore
> -------------------------------------
>
>                 Key: HBASE-18309
>                 URL: https://issues.apache.org/jira/browse/HBASE-18309
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: binlijin
>            Assignee: Reid Chan
>             Fix For: 3.0.0, 2.0.0-beta-1
>
>         Attachments: HBASE-18309.master.001.patch, HBASE-18309.master.002.patch, HBASE-18309.master.004.patch,
HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, HBASE-18309.master.007.patch,
HBASE-18309.master.008.patch, HBASE-18309.master.009.patch, HBASE-18309.master.010.patch,
HBASE-18309.master.011.patch, HBASE-18309.master.012.patch, space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.
> What's more, currently there's only one thread iterating the archive directory, and we
could use multiple threads cleaning sub directories in parallel to speed it up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message