hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tanping Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-1773) Remove a datanode from cluster if include list is not empty and this datanode is removed from both include and exclude lists
Date Tue, 22 Mar 2011 05:40:05 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tanping Wang updated HDFS-1773:
-------------------------------

    Attachment: HDFS-1773.patch

The problem was that after a datanode is registered with a namenode, it is
kept in the datanodeMap and not removed until it re-registrasted to serve a
different storageID( e.g. data storage is formatted on this datanode).  This
patch removes the datanode from the datanodeMap after a dfsadmin -refreshNodes
command, if both of the following two conditions are satisfied:
1) include list is not empty ( this essentially is a prerequisite.)
2) datanode does not appear in both include and exclude list.

This patch is to particularly serve the request from Ops that one
datanode was in the include list and removing it from both include and
exclude lists should make the datanode disappear from both live and dead node
list.   The include list is not empty is required.  If
include list is empty, that means all the datanodes are allowed to get registered, even if
a datanode is not in include and exclude list, it still can get registered
with the cluster.  Therefore, it is kept in the datanodeMap.

Manually tested with couple different scenarios:
Scenario I:
1) include two hosts, H1 and H2 in include file, start cluster
2) Add H2 to exclude file.  Remove H2 from include file, run dfsadmin -refreshNodes
Result: check web console that H2 disappeared.  H1 in live list.

Scenario II:
Every steps are the same as scenario I except in step2, still keep H2 in
include file.
Result: check web console that H2 disappeared.  H1 in live list.

Scenario III:
1) leave include and exclude list empty. Start cluster.
2) add H2 into exclude list, refresh nodes
2) remove H2 from exclude list, refresh nodes.
Result:  H2 should STILL in the dead node list.  Include list is empty, any
nodes can join.  Even remove H2 from exclude list, H2 is not considered to be
completely removed from the cluster.  So we do not remove H2 from datanodeMap.  Hence, it
appears in dead node list
still. 


> Remove a datanode from cluster if include list is not empty and this datanode is removed
from both include and exclude lists
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1773
>                 URL: https://issues.apache.org/jira/browse/HDFS-1773
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 0.20.203.1
>         Environment: branch-20-security
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Minor
>             Fix For: 0.20.4
>
>         Attachments: HDFS-1773.patch
>
>
> Our service engineering team who operates the clusters on a daily basis founds it is
confusing that after a data node is decommissioned, there is no way to make the cluster forget
about this data node and it always remains in the dead node list.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message