hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nandakumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10206) getBlockLocations might not sort datanodes properly by distance
Date Tue, 29 Nov 2016 06:26:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704378#comment-15704378

Nandakumar commented on HDFS-10206:

bq. NetworkTopology can be used by HDFS, YARN and MAPREDUCE. It is better to make variable
names more general.
Can you point out the variables which are to be made more generic? 

bq. But the reader should pick the closest one, either "Same Node" and "DataNode in same rack".
Perhaps you can clarify the setup.

Benchmarking was done with default replication factor - 3 (two replicas will be in same rack
as the writer and one will be in a different rack datanode)
{{NetworkTopology.sortByDistance}} method will call {{NetworkTopology.getWeight}} for every
replica of the block (within activeLen). Out of three, at least one of the replica will be
in off rack datanode even for "Same Node" and "DataNode in same rack". With 000.patch the
weight is calculated using network location for off rack datanodes which impacts the micro-benchmark
Sorry if I have confused you more.

So the weight value definition has changed. It should be fine given it isn't a public interface.
Still NetworkTopologyWithNodeGroup has its own getWeight definition based on the old definition.
Either we update that or keep the weight value.

According to {{NetworkTopologyWithNodeGroup.getWeight}}

0 for same node
1 for same group
2 for same rack
3 for off rack 

it aligns with weight definition of this patch, with an additional intermediate level (1 for
same group)

0 for same node
2 for same rack

for off rack in {{NetworkTopologyWithNodeGroup.getWeight}} we can call {{super.getWeight}}
which will calculate the weight using new logic rather than returning 3 for all the off rack

> getBlockLocations might not sort datanodes properly by distance
> ---------------------------------------------------------------
>                 Key: HDFS-10206
>                 URL: https://issues.apache.org/jira/browse/HDFS-10206
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Nandakumar
>         Attachments: HDFS-10206.000.patch, HDFS-10206.001.patch, HDFS-10206.002.patch
> If the DFSClient machine is not a datanode, but it shares its rack with some datanodes
of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}} might not put the local-rack
datanodes at the beginning of the sorted list. That is because the function didn't call {{networktopology.add(client);}}
to properly set the node's parent node; something required by {{networktopology.sortByDistance}}
to compute distance between two nodes in the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only distinguishes local
rack from remote rack, but it doesn't support general distance calculation to tell how remote
the rack is.
> {noformat}
> NetworkTopology.java
>   protected int getWeight(Node reader, Node node) {
>     // 0 is local, 1 is same rack, 2 is off rack
>     // Start off by initializing to off rack
>     int weight = 2;
>     if (reader != null) {
>       if (reader.equals(node)) {
>         weight = 0;
>       } else if (isOnSameRack(reader, node)) {
>         weight = 1;
>       }
>     }
>     return weight;
>   }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to address another
issue. Regardless of where we do the sorting, we still need fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object used by
DatanodeManager and requires Nodes stored in the topology to be {{DatanodeDescriptor}} for
block placement. So we need to make sure we don't pollute the  NetworkTopology if we plan
to fix it on the server side.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message