hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5037) Active NN should trigger its own edit log rolls
Date Fri, 01 Nov 2013 22:45:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811729#comment-13811729
] 

Andrew Wang commented on HDFS-5037:
-----------------------------------

I realize I should have waited for a Jenkins +1 before committing this, my mistake. I'm not
sure why the latest patch failed to apply, since it works for me locally. There's a virtual
Jenkins +1 on the v4 patch though and the changes in v5 were minor. I'll make watching the
next few trunk builds my responsibility to make sure nothing broke.

> Active NN should trigger its own edit log rolls
> -----------------------------------------------
>
>                 Key: HDFS-5037
>                 URL: https://issues.apache.org/jira/browse/HDFS-5037
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, namenode
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Todd Lipcon
>            Assignee: Andrew Wang
>            Priority: Critical
>             Fix For: 2.2.1
>
>         Attachments: hdfs-5037-1.patch, hdfs-5037-2.patch, hdfs-5037-3.patch, hdfs-5037-4.patch,
hdfs-5037-5.patch
>
>
> We've seen cases where the SBN/2NN went down, and then users accumulated very very large
edit log segments. This causes a slow startup time because the last edit log segment must
be read fully to recover it before the NN can start up again. Additionally, in the case of
QJM, it can trigger timeouts on recovery or edit log syncing because the very-large segment
has to get processed within a certain time bound.
> We could easily improve this by having the NN trigger its own edit log rolls on a configurable
size (eg every 256MB)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message