hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1820) [hbase] regionserver creates hlogs without bound
Date Thu, 30 Aug 2007 20:18:31 GMT
[hbase] regionserver creates hlogs without bound

                 Key: HADOOP-1820
                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
             Project: Hadoop
          Issue Type: Improvement
          Components: contrib/hbase
            Reporter: stack
            Priority: Minor

Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying
state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun
to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period,
the log is rotated and a new one is opened.  While the region server is up, the logs grow
in number without bound.  Only the most recent contain unpersisted edits.  If the region server
goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery,
the logs of edits are sorted and split  per region.  Recovery would run faster if it did not
have to plough through reams of stale edits.

Just now, I had a host crash w/ 112 log files each of 30k plus edits each.

We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could
check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their
last contained edit.  The thread could on a period ask each hosted region for the "lowest
highest" sequence id of all regions deployed.  Once this number had crossed out that on a
particular log, the log could be cleaned up safely.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message