Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 12 Dec 2013 21:51:10 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12678638.1384197172233.26854.1386885070025@arcas>
In-Reply-To: <JIRA.12678638.1384197172233@arcas>
References: <JIRA.12678638.1384197172233@arcas>
Subject: [jira] [Commented] (HDFS-5496) Make replication queue
 initialization asynchronous
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846783#comment-13846783 ] 

Kihwal Lee commented on HDFS-5496:
----------------------------------

I just want to provide a data point regarding the number of blocks to process/iteration. According to a measurement against a name node with a big name space, the initialization throughput was a little over 300K blocks/second. On this machine, the default limit of 10K will be equivalent to about 3.3ms of write lock duration.  There is a pressure to scan all blocks as soon as possible to avoid data loss by delayed replication, so a higher limit may be preferred in some situations. But I think it is okay as a default value.

> Make replication queue initialization asynchronous
> --------------------------------------------------
>
>                 Key: HDFS-5496
>                 URL: https://issues.apache.org/jira/browse/HDFS-5496
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Kihwal Lee
>            Assignee: Vinay
>         Attachments: HDFS-5496.patch, HDFS-5496.patch
>
>
> Today, initialization of replication queues blocks safe mode exit and certain HA state transitions. For a big name space, this can take hundreds of seconds with the FSNamesystem write lock held.  During this time, important requests (e.g. initial block reports, heartbeat, etc) are blocked.
> The effect of delaying the initialization would be not starting replication right away, but I think the benefit outweighs. If we make it asynchronous, the work per iteration should be limited, so that the lock duration is capped. 
> If full/incremental block reports and any other requests that modifies block state properly performs replication checks while the blocks are scanned and the queues populated in background, every block will be processed. (Some may be done twice)  The replication monitor should run even before all blocks are processed.
> This will allow namenode to exit safe mode and start serving immediately even with a big name space. It will also reduce the HA failover latency.


--
This message was sent by Atlassian JIRA
(v6.1.4#6159)