hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5496) Make replication queue initialization asynchronous
Date Wed, 11 Dec 2013 17:53:09 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845580#comment-13845580
] 

Kihwal Lee commented on HDFS-5496:
----------------------------------

The following change would have been fine if leaving safe mode and initializing replication
queues were synchronized.  It appears {{checkMode()}} can start a background initialization
before leaving the safe mode. Since the queues are unconditionally cleared right before the
following, an on-going initialization should be stopped and redone.

{code}
-        if (!isInSafeMode() ||
-            (isInSafeMode() && safeMode.isPopulatingReplQueues())) {
+        // We only need to reprocess the queue in HA mode and not in safemode
+        if (!isInSafeMode() && haEnabled) {
{code}

There have been discussions regarding removing safe mode extension and perhaps safe mode monitor.
That will make the check/logic simpler.

> Make replication queue initialization asynchronous
> --------------------------------------------------
>
>                 Key: HDFS-5496
>                 URL: https://issues.apache.org/jira/browse/HDFS-5496
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Kihwal Lee
>            Assignee: Vinay
>         Attachments: HDFS-5496.patch, HDFS-5496.patch
>
>
> Today, initialization of replication queues blocks safe mode exit and certain HA state
transitions. For a big name space, this can take hundreds of seconds with the FSNamesystem
write lock held.  During this time, important requests (e.g. initial block reports, heartbeat,
etc) are blocked.
> The effect of delaying the initialization would be not starting replication right away,
but I think the benefit outweighs. If we make it asynchronous, the work per iteration should
be limited, so that the lock duration is capped. 
> If full/incremental block reports and any other requests that modifies block state properly
performs replication checks while the blocks are scanned and the queues populated in background,
every block will be processed. (Some may be done twice)  The replication monitor should run
even before all blocks are processed.
> This will allow namenode to exit safe mode and start serving immediately even with a
big name space. It will also reduce the HA failover latency.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message