hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HDFS-13098) RBF: Datanodes interacting with Routers
Date Fri, 02 Feb 2018 18:22:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350767#comment-16350767

Íñigo Goiri commented on HDFS-13098:

bq. Do these subclusters also dynamically generated? That is say, we have N datanodes, then
these datanodes communicate with the Router. Based on the assignment policies, the Router
decides which subcluster the datanode belong to.
We could add new subclusters (i.e., new namenodes) and then we can use consistent hashing
to just move a small subset of the servers to the new subcluster.

bq. But if the machines are moving across subclusters, how we plan to do? Decommission in
original subclsuter then add into new subcluster?
For now we would do this change of subclusters for servers that are empty: new ones or servers
that get reimaged/decommissioned.
Once we support rebalancing, we could merge both things.
At the beginning, we can leverage regular HDFS federation and the DN could be in subcluster_old
and subcluster_new at the same time and then decommission it from subcluster_old.
In any case, the first version could prevent moving DNs out of the current subcluster.

> RBF: Datanodes interacting with Routers
> ---------------------------------------
>                 Key: HDFS-13098
>                 URL: https://issues.apache.org/jira/browse/HDFS-13098
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Íñigo Goiri
>            Priority: Major
> Datanodes talk to particular Namenodes. We could use the Router infrastructure for the
Datanodes to register/heartbeating into them and the Routers would forward this to particular
Namenodes. This would make the assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message