Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sun, 30 Apr 2017 22:22:07 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13024532.1480531160000.88341.1493590927521@Atlassian.JIRA>
In-Reply-To: <JIRA.13024532.1480531160000@Atlassian.JIRA>
References: <JIRA.13024532.1480531160000@Atlassian.JIRA> <JIRA.13024532.1480531160758@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17215) Separate small/large file delete
 threads in HFileCleaner to accelerate archived hfile cleanup speed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 30 Apr 2017 22:22:14 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990408#comment-15990408 ] 

Hudson commented on HBASE-17215:
--------------------------------

ABORTED: Integrated in Jenkins build HBase-HBASE-14614 #190 (See [https://builds.apache.org/job/HBase-HBASE-14614/190/])
HBASE-17215 Separate small/large file delete threads in HFileCleaner to (liyu: rev 9facfa550f1e7386be3a04d84f7e8013f5002965)
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
HBASE-17854 Use StealJobQueue in HFileCleaner after HBASE-17215 (liyu: rev cbcbcf4dcd3401327cc36173f3ca8e5362da1e0c)
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/StealJobQueue.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestStealJobQueue.java


> Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-17215
>                 URL: https://issues.apache.org/jira/browse/HBASE-17215
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0
>
>         Attachments: HBASE-17215.patch, HBASE-17215.v2.patch, HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we have per CF flush, we still have the {{hbase.regionserver.optionalcacheflushinterval}} setting and some other mechanism to avoid data kept in memory for too long to flush small hfiles. In our online environment we found the single thread cleaner kept cleaning earlier flushed small files while large files got no chance, which caused disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the workload of namenode, so here we propose to separate large/small hfile cleaner threads just like we do for compaction, and it turned out to work well in our cluster.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)