Return-Path: 
 <issues-return-112996-apmail-hbase-issues-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-issues-archive@www.apache.org
Delivered-To: apmail-hbase-issues-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF15010B28
	for <apmail-hbase-issues-archive@www.apache.org>;
 Thu, 29 Aug 2013 15:25:55 +0000 (UTC)
Received: (qmail 80533 invoked by uid 500); 29 Aug 2013 15:25:55 -0000
Delivered-To: apmail-hbase-issues-archive@hbase.apache.org
Received: (qmail 80468 invoked by uid 500); 29 Aug 2013 15:25:55 -0000
Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@hbase.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@hbase.apache.org>
List-Post: <mailto:issues@hbase.apache.org>
List-Id: <issues.hbase.apache.org>
Delivered-To: mailing list issues@hbase.apache.org
Received: (qmail 80287 invoked by uid 99); 29 Aug 2013 15:25:53 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 15:25:53 +0000
Date: Thu, 29 Aug 2013 15:25:53 +0000 (UTC)
From: "Dave Latham (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12663509.1376426662236.54590.1377789953398@arcas>
In-Reply-To: <JIRA.12663509.1376426662236@arcas>
References: <JIRA.12663509.1376426662236@arcas>
Subject: [jira] [Updated] (HBASE-9208) ReplicationLogCleaner slow at large
 scale
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HBASE-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dave Latham updated HBASE-9208:
-------------------------------

    Release Note: The FileCleanerDelegate interface has changed to operate on a batch of files at a time.  Any custom implementations that implemented the interface directly for cleaning HFile's of HLog's will have to be updated to implement the new interface or extend one of the included abstract base classes.

Thanks, JD, for the testing, and Stack for driving it in.  I added a release note.
                
> ReplicationLogCleaner slow at large scale
> -----------------------------------------
>
>                 Key: HBASE-9208
>                 URL: https://issues.apache.org/jira/browse/HBASE-9208
>             Project: HBase
>          Issue Type: Improvement
>          Components: Replication
>            Reporter: Dave Latham
>            Assignee: Dave Latham
>             Fix For: 0.98.0, 0.96.0
>
>         Attachments: HBASE-9208-0.94.patch, HBASE-9208-0.94-v2.patch, HBASE-9208.patch, HBASE-9208-v2.patch, HBASE-9208-v3.patch
>
>
> At a large scale the ReplicationLogCleaner fails to clean up .oldlogs as fast as the cluster is producing them.  For each old HLog file that has been replicated and should be deleted the ReplicationLogCleaner checks every replication queue in ZooKeeper before removing it.  This means that as a cluster scales up the number of files to delete scales as well as the time to delete each file so the cleanup chore scales quadratically.  In our case it reached the point where the oldlogs were growing faster than they were being cleaned up.
> We're now running with a patch that allows the ReplicationLogCleaner to refresh its list of files in the replication queues from ZooKeeper just once for each batch of files the CleanerChore wants to evaluate.
> I'd propose updating FileCleanerDelegate to take a List<FileStatus> rather than a single one at a time.  This would allow file cleaners that check an external resource for references such as ZooKeeper (for ReplicationLogCleaner) or HDFS (for SnapshotLogCleaner which looks like it may also have similar trouble at scale) to load those references once per batch rather than for every log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira