hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reid Chan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
Date Wed, 08 Nov 2017 09:49:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243622#comment-16243622

Reid Chan commented on HBASE-18309:

bq. Although both needs multiple threads, HFileCleaner separates the threads into two categories
for cleaning large/small hfiles and some StealJobQueue trick used there, while I don't think
we need the same logic for LogCleaner. I'd suggest a deeper look into the source if you still
like to unify the logic.
I get your idea, apparently we are in different directions now. You focus on the deletion
on files, but my implementation focus on speeding up the logic of deletion under oldWALs/archive.
I'd suggest you take a look at the patch.
I may misunderstand [~aoxiang]'s original intent, which is to focus on deletion files as {{HFileCleaner}}
does. But i'm sure my way will improve efficiency as well. Thanks [~carp84]

> Support multi threads in CleanerChore
> -------------------------------------
>                 Key: HBASE-18309
>                 URL: https://issues.apache.org/jira/browse/HBASE-18309
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: binlijin
>            Assignee: Reid Chan
>         Attachments: HBASE-18309.master.001.patch, HBASE-18309.master.002.patch
> There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.

This message was sent by Atlassian JIRA

View raw message