hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5060) NN should proactively perform a saveNamespace if it has a huge number of outstanding uncheckpointed transactions
Date Fri, 02 Aug 2013 19:19:52 GMT
Aaron T. Myers created HDFS-5060:
------------------------------------

             Summary: NN should proactively perform a saveNamespace if it has a huge number
of outstanding uncheckpointed transactions
                 Key: HDFS-5060
                 URL: https://issues.apache.org/jira/browse/HDFS-5060
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 2.1.0-beta
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers


In a properly-functioning HDFS system, checkpoints will be triggered either by the secondary
NN or standby NN regularly, by default every hour or 1MM outstanding edits transactions, whichever
come first. However, in cases where this second node is down for an extended period of time,
the number of outstanding transactions can grow so large as to cause a restart to take an
inordinately long time.

This JIRA proposes to make the active NN monitor its number of outstanding transactions and
perform a proactive local saveNamespace if it grows beyond a configurable threshold. I'm envisioning
something like 10x the configured number of transactions which in a properly-functioning cluster
would result in a checkpoint from the second NN. Though this would be disruptive to clients
while it's taking place, likely for a few minutes, this seems better than the alternative
of a subsequent multi-hour restart and should never actually occur in a properly-functioning
cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message