hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2737) HA: Automatically trigger log rolls periodically on the active NN
Date Fri, 20 Jan 2012 01:27:40 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aaron T. Myers updated HDFS-2737:

    Attachment: HDFS-2737-HDFS-1623.patch

Here's an update to Todd's last patch. Not much has changed. It's rebased on the tip of HDFS-1623,
and has some cleanup, including:

# Changing the config option units from milliseconds to seconds.
# Improved logging in a few places.
# s/interval/period/g in the new config options, to be consistent with the checkpoint options.
# Roll the edit log before trying to tail the logs, which should keep the standby a little

In addition to the unit test included in the patch, I also did some manual cluster testing,
which worked very well.
> HA: Automatically trigger log rolls periodically on the active NN
> -----------------------------------------------------------------
>                 Key: HDFS-2737
>                 URL: https://issues.apache.org/jira/browse/HDFS-2737
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>         Attachments: HDFS-2737-HDFS-1623.patch, hdfs-2737-prelim.txt
> Currently, the edit log tailing process can only read finalized log segments. So, if
the active NN is not rolling its logs periodically, the SBN will lag a lot. This also causes
many datanode messages to be queued up in the PendingDatanodeMessage structure.
> To combat this, the active NN needs to roll its logs periodically -- perhaps based on
a time threshold, or perhaps based on a number of transactions. I'm not sure yet whether it's
better to have the NN roll on its own or to have the SBN ask the active NN to roll its logs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message