hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
Date Thu, 28 Dec 2017 04:02:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305035#comment-16305035

stack commented on HBASE-18309:

[~reidchan] It runs too often and the spew in a big cluster is profuse. I don't think we should
log a message unless there is an error. I'm trying to figure whey we get the "Couldn't delete...
" messages. If not an error, I'll suppress these lines too.

Running w/ all the CPUs in the machine verges on the abusive don't you think? Should we do
less.  1/4? 1/8th?


> Support multi threads in CleanerChore
> -------------------------------------
>                 Key: HBASE-18309
>                 URL: https://issues.apache.org/jira/browse/HBASE-18309
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: binlijin
>            Assignee: Reid Chan
>             Fix For: 3.0.0, 2.0.0-beta-1
>         Attachments: HBASE-18309.addendum.patch, HBASE-18309.master.001.patch, HBASE-18309.master.002.patch,
HBASE-18309.master.004.patch, HBASE-18309.master.005.patch, HBASE-18309.master.006.patch,
HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, HBASE-18309.master.009.patch,
HBASE-18309.master.010.patch, HBASE-18309.master.011.patch, HBASE-18309.master.012.patch,
> There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find
this is not enough. The number of files under oldWALs reach the max-directory-items limit
of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash
not happened any more.
> What's more, currently there's only one thread iterating the archive directory, and we
could use multiple threads cleaning sub directories in parallel to speed it up.

This message was sent by Atlassian JIRA

View raw message