hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-8599) HLogs in ZK are not cleaned up when replication lag is minimal
Date Thu, 23 May 2013 23:15:20 GMT

     [ https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Varun Sharma updated HBASE-8599:
--------------------------------

    Attachment: 8599-trunk.patch
    
> HLogs in ZK are not cleaned up when replication lag is minimal
> --------------------------------------------------------------
>
>                 Key: HBASE-8599
>                 URL: https://issues.apache.org/jira/browse/HBASE-8599
>             Project: HBase
>          Issue Type: Improvement
>          Components: Replication
>    Affects Versions: 0.98.0, 0.94.7
>            Reporter: Varun Sharma
>         Attachments: 8599-0.94.patch, 8599-trunk.patch
>
>
> On a cluster with very low replication lag (as measured by ageOfLastShippedOp on source),
we found HLogs accumulating and not being cleaned up as new WAL(s) are rolled.
> Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever the current
WAL is not being written to any more - as suggested by currentWALBeingWrittenTo being false.
However, when lags are small, we may hit the following block first and continue onto the next
WAL without clearing the old WAL(s)...
> ReplicationSource::run() {
>     if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo = false)) {
>         // If we are here, then we advance to the next WAL without any cleaning
>         // and close existing WAL
>         continue;
>     }
>     // Ship some edits and call logPositionAndCleanOldLogs
> }
> If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older logs are
not cleaned out and persist in the zookeeper node since we simply call "continue" and skip
the subsequent logPositionAndCleanOldLogs call - if its called more than once, we do end up
clearing the old logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message