hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reid Chan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18309) Support multi threads to clean oldWALs
Date Tue, 07 Nov 2017 12:30:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241929#comment-16241929

Reid Chan commented on HBASE-18309:

To trigger CleanerChore, i created some directories and files (not exactly hbase formatted
file) under ${hbase.rootdir}/archive (or ${hbase.rootdir}/oldWALs is the same, and ${hbase.rootdir}
is /tmp/hbase in my case).
More specifically, 10 directories under /tmp/hbase/archive, and in each directory, there're
100 files and subdirectories randomly generated, like /tmp/hbase/archive/folder1/file_0, /tmp/hbase/archive/folder1/folder2/folder3/file_x,
/tmp/hbase/archive/folder1/folder4, and so on. Then CleanerChore is supposed to clean them
up, and time is measured. Repeat it 10 times.
I hope i make it clear.
Following is the result tables (in ms):
||Test case||avg||min||max||
|single threads with sorting|1282.5|1089|1982|
|4 threads with sorting|1043.6|718|1359|
|4 threads without sorting|313.3|215|489|
|2 threads without sorting|310.5|204|434|

> Support multi threads to clean oldWALs
> --------------------------------------
>                 Key: HBASE-18309
>                 URL: https://issues.apache.org/jira/browse/HBASE-18309
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: binlijin
>            Assignee: Reid Chan
>         Attachments: HBASE-18309.master.001.patch
> There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.

This message was sent by Atlassian JIRA

View raw message