hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Jadhav (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
Date Thu, 26 Jan 2017 16:33:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839978#comment-15839978

Ajay Jadhav commented on HBASE-17280:

I see some merge conflicts for the master branch. 
So, I'll add the master patch soon. I have attached the branch-1.2 patch.

Actually, the patch is based on branch 1.2.2. There I saw the following cleaners in HMaster:
  private ReplicationZKLockCleanerChore replicationZKLockCleanerChore;
  private LogCleaner logCleaner;
  private HFileCleaner hfileCleaner;

Our main intention with this patch was to avoid running the expensive cleaner chore during
peak times. During our experimentation, we saw a lot of HFiles and WAL log related files getting
created inside archive dir (didn't see ZKlock related files). Since we were replacing hdfs
with S3, these delete calls will take forever to complete.

As for setCleanerRunning(): We want to enable/ disable the cleaner entirely.  So, when the
user run cleaner_switch, we enable/ disable both the cleaners. They both go down/ up together.

Both HFileCleaner and LogCleaner are implementing cleanerchore which is doing this expensive
list and delete files from the dir which we want to avoid due to underlying filesystem. 
ReplicationZKLockCleanerChore seems like is trying to clean replication locks on zk and directly
implements the scheduledchore.

> Add mechanism to control hbase cleaner behavior
> -----------------------------------------------
>                 Key: HBASE-17280
>                 URL: https://issues.apache.org/jira/browse/HBASE-17280
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client, hbase, shell
>    Affects Versions: 2.0.0, 1.2.0
>            Reporter: Ajay Jadhav
>            Priority: Minor
>             Fix For: 2.0.0, 1.2.0
>         Attachments: HBASE-17280.branch-1.2.patch, HBASE-17280.branch-2.0.patch, HBASE-17280.v1-branch-1.2.patch,
HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch
> Cleaner is used to get rid of archived HFiles and old WALs in HBase.
> In the case of heavy workload, cleaner can affect query performance by creating a lot
of connections to perform costly reads/ writes against underlying filesystem.
> This patch allows the user to control HBase cleaner behavior by providing shell commands
to enable/ disable and manually run it.

This message was sent by Atlassian JIRA

View raw message