hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-14563) Enhance interface about recommissioning/decommissioning
Date Wed, 19 Jun 2019 20:58:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867999#comment-16867999
] 

Íñigo Goiri edited comment on HDFS-14563 at 6/19/19 8:57 PM:
-------------------------------------------------------------

{quote}
currently refreshNodes call is dispatched to all NNs in a cluster, while with the centralized
approach we may want one NN to perform the action and populate the changes to all other NNs?
{quote}
I would keep the same behavior. For refreshNodes it is just the same:
* NFS Option: user can change the file by hand, invokes refreshNodes in all Namenodes, each
Namenodes reads from the file and does the loading.
* ZK Option: user invokes refreshNodes in all Namenodes, each Namenodes reads from ZK and
does the loading.

The new API is the one that has the different behavior.
The user invokes the new RPC request (e.g., setNodeState) in the active Namenode, the NN updates
the store (e.g., NFS or ZK), it udpates the local DN mapping.
At this point, the NN (or the user) should potentially call refresNodes in the other Namenodes
that will take the changes from ZK or NFS.


was (Author: elgoiri):
{quote}
currently refreshNodes call is dispatched to all NNs in a cluster, while with the centralized
approach we may want one NN to perform the action and populate the changes to all other NNs?
{quote}
I would keep the same behavior. For refreshNodes it is just the same:
** NFS Option: user can change the file by hand, invokes refreshNodes in all Namenodes, each
Namenodes reads from the file and does the loading.
** ZK Option: user invokes refreshNodes in all Namenodes, each Namenodes reads from ZK and
does the loading.

The new API is the one that has the different behavior.
The user invokes the new RPC request (e.g., setNodeState) in the active Namenode, the NN updates
the store (e.g., NFS or ZK), it udpates the local DN mapping.
At this point, the NN (or the user) should potentially call refresNodes in the other Namenodes
that will take the changes from ZK or NFS.

> Enhance interface about recommissioning/decommissioning
> -------------------------------------------------------
>
>                 Key: HDFS-14563
>                 URL: https://issues.apache.org/jira/browse/HDFS-14563
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client, namenode
>            Reporter: He Xiaoqiao
>            Assignee: He Xiaoqiao
>            Priority: Major
>         Attachments: HDFS-14563.001.patch, HDFS-14563.002.patch, mt_mode-2.txt
>
>
> In current implementation, if we need to decommissioning or recommissioning one datanode,
the only way is add the datanode to include or exclude file under namenode configuration path
then execute command `bin/hadoop dfsadmin -refreshNodes` and trigger namenode to reload include/exclude
and start to recommissioning or decommissioning datanode.
> The shortcomings of this approach is that:
> a. namenode reload include/exclude configuration file from devices, if I/O load is high,
handler may be blocked.
> b. namenode has to process every datnodes in include and exclude configurations, if there
are many datanodes (very common for large cluster) pending to process, namenode will be hung
for hundred seconds to wait recommision/decommision finish at the worst since holding write
lock.
> I think we should expose one lightweight interface to support recommissioning or decommissioning
single datanode, thus we can operate datanode using dfsadmin more smooth.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message