hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5037) Active NN should trigger its own edit log rolls
Date Fri, 01 Nov 2013 13:42:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811253#comment-13811253

Aaron T. Myers commented on HDFS-5037:

I'm pretty confident that the eclipse failure is unrelated:

[ERROR] Failed to execute goal org.codehaus.mojo:build-helper-maven-plugin:1.5:add-source
(add-jsp-generated-sources-directory) on project hadoop-hdfs: Execution add-jsp-generated-sources-directory
of goal org.codehaus.mojo:build-helper-maven-plugin:1.5:add-source failed: Unable to load
the mojo 'add-source' in the plugin 'org.codehaus.mojo:build-helper-maven-plugin:1.5'. A required
class is missing: org.codehaus.mojo.buildhelper.AddSourceMojo

A few small comments:

# One too many 'a's: "+      // # edit autoroll threshold is a a multiple of the checkpoint
threshold "
# Is there some reason that the variables in the TestEditLogAutoroll class need to be static?
Seems like they should be instance variables to me.
# Doesn't look like you ever use the nn1 variable in the test.
# Similarly you only ever use the namesystem variable once, just to get a reference to the
edit log. Might want to just chain those calls.
# Are we positive that the test would fail without the auto roller in place? i.e. is it not
possible that the standby NN in the minicluster might trigger an edit log roll and cause the
test to pass without the auto roller having done anything?

+1 once these are addressed.

> Active NN should trigger its own edit log rolls
> -----------------------------------------------
>                 Key: HDFS-5037
>                 URL: https://issues.apache.org/jira/browse/HDFS-5037
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, namenode
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Todd Lipcon
>            Assignee: Andrew Wang
>            Priority: Critical
>         Attachments: hdfs-5037-1.patch, hdfs-5037-2.patch, hdfs-5037-3.patch, hdfs-5037-4.patch
> We've seen cases where the SBN/2NN went down, and then users accumulated very very large
edit log segments. This causes a slow startup time because the last edit log segment must
be read fully to recover it before the NN can start up again. Additionally, in the case of
QJM, it can trigger timeouts on recovery or edit log syncing because the very-large segment
has to get processed within a certain time bound.
> We could easily improve this by having the NN trigger its own edit log rolls on a configurable
size (eg every 256MB)

This message was sent by Atlassian JIRA

View raw message