hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5496) Make replication queue initialization asynchronous
Date Mon, 16 Dec 2013 12:40:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849110#comment-13849110
] 

Hadoop QA commented on HDFS-5496:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12618878/HDFS-5496.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5730//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5730//console

This message is automatically generated.

> Make replication queue initialization asynchronous
> --------------------------------------------------
>
>                 Key: HDFS-5496
>                 URL: https://issues.apache.org/jira/browse/HDFS-5496
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Kihwal Lee
>            Assignee: Vinay
>         Attachments: HDFS-5496.patch, HDFS-5496.patch, HDFS-5496.patch, HDFS-5496.patch
>
>
> Today, initialization of replication queues blocks safe mode exit and certain HA state
transitions. For a big name space, this can take hundreds of seconds with the FSNamesystem
write lock held.  During this time, important requests (e.g. initial block reports, heartbeat,
etc) are blocked.
> The effect of delaying the initialization would be not starting replication right away,
but I think the benefit outweighs. If we make it asynchronous, the work per iteration should
be limited, so that the lock duration is capped. 
> If full/incremental block reports and any other requests that modifies block state properly
performs replication checks while the blocks are scanned and the queues populated in background,
every block will be processed. (Some may be done twice)  The replication monitor should run
even before all blocks are processed.
> This will allow namenode to exit safe mode and start serving immediately even with a
big name space. It will also reduce the HA failover latency.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message