Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 23 May 2013 23:17:20 +0000 (UTC)
From: "Varun Sharma (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12649155.1369332911345.14746.1369351040986@arcas>
In-Reply-To: <JIRA.12649155.1369332911345@arcas>
References: <JIRA.12649155.1369332911345@arcas>
Subject: [jira] [Updated] (HBASE-8599) HLogs in ZK are not cleaned up when
 replication lag is minimal
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Varun Sharma updated HBASE-8599:
--------------------------------

    Attachment:     (was: 8599-trunk.patch)
    
> HLogs in ZK are not cleaned up when replication lag is minimal
> --------------------------------------------------------------
>
>                 Key: HBASE-8599
>                 URL: https://issues.apache.org/jira/browse/HBASE-8599
>             Project: HBase
>          Issue Type: Improvement
>          Components: Replication
>    Affects Versions: 0.98.0, 0.94.7
>            Reporter: Varun Sharma
>             Fix For: 0.98.0, 0.94.9
>
>         Attachments: 8599-0.94.patch
>
>
> On a cluster with very low replication lag (as measured by ageOfLastShippedOp on source), we found HLogs accumulating and not being cleaned up as new WAL(s) are rolled.
> Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever the current WAL is not being written to any more - as suggested by currentWALBeingWrittenTo being false. However, when lags are small, we may hit the following block first and continue onto the next WAL without clearing the old WAL(s)...
> ReplicationSource::run() {
>     if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo = false)) {
>         // If we are here, then we advance to the next WAL without any cleaning
>         // and close existing WAL
>         continue;
>     }
>     // Ship some edits and call logPositionAndCleanOldLogs
> }
> If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older logs are not cleaned out and persist in the zookeeper node since we simply call "continue" and skip the subsequent logPositionAndCleanOldLogs call - if its called more than once, we do end up clearing the old logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira