hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5037) Active NN should trigger its own edit log rolls
Date Fri, 26 Jul 2013 16:47:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720946#comment-13720946
] 

Harsh J commented on HDFS-5037:
-------------------------------

We're discussing rolling only the edit log, i.e. an inbuilt automated {{hdfs dfsadmin -rollEdits}}.
This doesn't have any memory repercussions. The goal being not to end up with a huge edits-inprogress
file that takes hours to transfer by QJM when syncing logs (as one case).
                
> Active NN should trigger its own edit log rolls
> -----------------------------------------------
>
>                 Key: HDFS-5037
>                 URL: https://issues.apache.org/jira/browse/HDFS-5037
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, namenode
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Todd Lipcon
>
> We've seen cases where the SBN/2NN went down, and then users accumulated very very large
edit log segments. This causes a slow startup time because the last edit log segment must
be read fully to recover it before the NN can start up again. Additionally, in the case of
QJM, it can trigger timeouts on recovery or edit log syncing because the very-large segment
has to get processed within a certain time bound.
> We could easily improve this by having the NN trigger its own edit log rolls on a configurable
size (eg every 256MB)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message