hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2794) HA: Active NN may purge edit log files before standby NN has a chance to read them
Date Mon, 06 Feb 2012 06:45:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201082#comment-13201082
] 

Todd Lipcon commented on HDFS-2794:
-----------------------------------

Attached patch adds a new configuration, dfs.namenode.num.extra.edits.retained, which causes
the NN to not purge a given number of edits that are older than the oldest retained local
checkpoint. This seemed preferable to me than the other option discussed, which was to configure
the NN to retain many more images. The reason is that even a million edits (the default) would
be on the order of a few hundred MB, whereas retaining a day's worth of checkpoints might
be on the order of hundreds of GB for a large cluster making frequent checkpoints.

Retaining edits for a long period of time has some other useful applications, as well (eg
a binary form of audit log).
                
> HA: Active NN may purge edit log files before standby NN has a chance to read them
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-2794
>                 URL: https://issues.apache.org/jira/browse/HDFS-2794
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Aaron T. Myers
>            Assignee: Todd Lipcon
>         Attachments: hdfs-2794.txt
>
>
> Given that the active NN is solely responsible for purging finalized edit log segments,
and given that the active NN has no way of knowing when the standby reads edit logs, it's
 possible that the standby NN could fail to read all edits it needs before the active purges
the files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message