hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8599) HLogs in ZK are not cleaned up when replication lag is minimal
Date Wed, 26 Jun 2013 04:17:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693619#comment-13693619
] 

stack commented on HBASE-8599:
------------------------------

+1 on patch.

[~jdcryans] or [~ctrezzo], what you fellas think?
                
> HLogs in ZK are not cleaned up when replication lag is minimal
> --------------------------------------------------------------
>
>                 Key: HBASE-8599
>                 URL: https://issues.apache.org/jira/browse/HBASE-8599
>             Project: HBase
>          Issue Type: Improvement
>          Components: Replication
>    Affects Versions: 0.98.0, 0.94.7
>            Reporter: Varun Sharma
>            Assignee: Varun Sharma
>             Fix For: 0.98.0, 0.94.10
>
>         Attachments: 8599-0.94.patch, 8599-trunk.patch, 8599-trunk-v2.patch
>
>
> On a cluster with very low replication lag (as measured by ageOfLastShippedOp on source),
we found HLogs accumulating and not being cleaned up as new WAL(s) are rolled.
> Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever the current
WAL is not being written to any more - as suggested by currentWALBeingWrittenTo being false.
However, when lags are small, we may hit the following block first and continue onto the next
WAL without clearing the old WAL(s)...
> ReplicationSource::run() {
>     if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo = false)) {
>         // If we are here, then we advance to the next WAL without any cleaning
>         // and close existing WAL
>         continue;
>     }
>     // Ship some edits and call logPositionAndCleanOldLogs
> }
> If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older logs are
not cleaned out and persist in the zookeeper node since we simply call "continue" and skip
the subsequent logPositionAndCleanOldLogs call - if its called more than once, we do end up
clearing the old logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message