hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Yan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13098) RBF: Datanodes interacting with Routers
Date Thu, 01 Feb 2018 17:47:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348992#comment-16348992

Wei Yan commented on HDFS-13098:

{quote}Currently, we do the assignment of DNs to subclusters using external tools that generate {{hdfs-site.xml}}.
These tools could be moved into the RBF infrastructure.

To solve this, we could make the DNs to just register the first time through the Router and
afterwards switch to heartbeating into the actual Namenodes.
If I understand correctly, here the first hearbeat to Routers is to determine which subcluster
this new DN will join. Afterwards, the DN will only talk to NNs in that subcluster. If this's
correct, the first heartbeat to Routers is just like an "external" tool to generate the hdfs-site.xml,
{quote} * Policies to assign datanodes to subcluters
 * Datanodes to make blockpool configuration dynamic{quote}
Do u have some example policies here, to determine the DN assignment? I'm thinking about data
assignment policies, like allocating data to subcluter to balance subcluster workload, moving
"warm" data to cheaper hardware subcluster, etc. Maybe I miss sth here, but do we need to
relocate DNs from clusters to clusters quite often?

> RBF: Datanodes interacting with Routers
> ---------------------------------------
>                 Key: HDFS-13098
>                 URL: https://issues.apache.org/jira/browse/HDFS-13098
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Íñigo Goiri
>            Priority: Major
> Datanodes talk to particular Namenodes. We could use the Router infrastructure for the
Datanodes to register/heartbeating into them and the Routers would forward this to particular
Namenodes. This would make the assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message