hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager
Date Fri, 19 Dec 2014 23:21:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254213#comment-14254213
] 

Ming Ma commented on HDFS-7411:
-------------------------------

Couple more comments:

* dfs.namenode.decommission.blocks.per.node together with dfs.namenode.decommission.nodes.per.interval
can control how long DecommissionManager will hold FSN write lock. Should we just define dfs.namenode.decommission.blocks.per.interval
instead? If dfs.namenode.decommission.blocks.per.interval is defined in the configuration,
use it. If only dfs.namenode.decommission.nodes.per.interval is defined, use the run time
"average block count per node" to estimate.

* It seems exceededNumBlocksPerCheck is called only when DecommissionManager moves to to check
the next DN. If a DN has lots of blocks, the check won't stop earlier.

* This patch has make "dfsadmin -refreshNodes" asynchronously for decommission; e.g. startDecommission
no longer calls checkDecommission which takes FSN write lock. But it has to wait for DecommissionManager's
next check to kick off the replication process. https://issues.apache.org/jira/browse/HDFS-5757
and https://issues.apache.org/jira/browse/HDFS-7521 provide asynchronous notification so replication
can start right away.

* We can also make "dfsadmin -refreshNodes" asynchronously for recommission. But it doesn't
have be done by this jira. We can let other jiras handle this.

> Refactor and improve decommissioning logic into DecommissionManager
> -------------------------------------------------------------------
>
>                 Key: HDFS-7411
>                 URL: https://issues.apache.org/jira/browse/HDFS-7411
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.5.1
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: hdfs-7411.001.patch, hdfs-7411.002.patch, hdfs-7411.003.patch, hdfs-7411.004.patch,
hdfs-7411.005.patch
>
>
> Would be nice to split out decommission logic from DatanodeManager to DecommissionManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message