hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning
Date Wed, 12 Jun 2019 16:07:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862241#comment-16862241
] 

Kihwal Lee commented on HDFS-14563:
-----------------------------------

In our experience, decommissioning and adding new nodes are manageable and the latency is
okay even for 5,000+ node clusters. We avoid recommissioning as it is expensive. It is much
better to wipe out and add as a new node.

In any case, command-based approach has merits. This also applies to maintenance mode, if
you look at 2.9 and greater. [~rajive] once stated that an admin command to start/end maintenance
mode would be preferred from the service engineering point of view. It was not further pursued
because of non-persistent nature of the state changes results from such commands. You will
have to address this to make this happen.  Many have existing automation built around the
existing include/exclude/refresh logic. Whatever the new way is, it will be desirable to have
something easier to integrate from  or migrate from the existing approach.

> Enhance interface about recommissioning/decommissioning
> -------------------------------------------------------
>
>                 Key: HDFS-14563
>                 URL: https://issues.apache.org/jira/browse/HDFS-14563
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client, namenode
>            Reporter: He Xiaoqiao
>            Assignee: He Xiaoqiao
>            Priority: Major
>
> In current implementation, if we need to decommissioning or recommissioning one datanode,
the only way is add the datanode to include or exclude file under namenode configuration path
then execute command `bin/hadoop dfsadmin -refreshNodes` and trigger namenode to reload include/exclude
and start to recommissioning or decommissioning datanode.
> The shortcomings of this approach is that:
> a. namenode reload include/exclude configuration file from devices, if I/O load is high,
handler may be blocked.
> b. namenode has to process every datnodes in include and exclude configurations, if there
are many datanodes (very common for large cluster) pending to process, namenode will be hung
for hundred seconds to wait recommision/decommision finish at the worst since holding write
lock.
> I think we should expose one lightweight interface to support recommissioning or decommissioning
single datanode, thus we can operate datanode using dfsadmin more smooth.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message