hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nandakumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10206) getBlockLocations might not sort datanodes properly by distance
Date Sat, 19 Nov 2016 20:21:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679807#comment-15679807
] 

Nandakumar commented on HDFS-10206:
-----------------------------------

{quote}
Here is another option. DatanodeManager#sortLocatedBlock already knows if its a datanode.
So we can have a new NetworkTopology#sortByDistance that supports check-by-reference.
{quote}
This option looks better, thanks for the suggestion [~mingma]. Will make the changes accordingly
and upload a new patch.

In mean time I was able to do micro benchmark with [^HDFS-10206.000.patch], the benchmarks
only measures the time taken by {{NetworkTopology.sortByDistance}}

HDFS Version: 2.7.1
Reading a single block file. (file size: 120 MB)
The values are in nanosecond.

*Without patch*

||Client on|| Run 1 || Run 2 || Run 3 || Run 4 || Run 5 ||
|Same Node| 99710 | 124014 | 134857 | 146936 | 111543 |
|DataNode in same rack| 169122 | 99805 | 124058 | 134566 | 269096 |
|DataNode in different rack| 114552 | 103003 | 153313 | 92008 | 114279 |
|Non-DataNode in same rack| 97960 | 199611 | 77948 | 101324 | 90920 |
|Non-DataNode in different rack| 93002 | 182436 | 104600 | 96434 | 138167 |

*With patch*

||Client on|| Run 1 || Run 2 || Run 3 || Run 4 || Run 5 ||
|Same Node| 121510 | 185741 | 110382 | 180451 | 132131 |
|DataNode in same rack| 182892 | 128597 | 187518 | 136754 | 385739 |
|DataNode in different rack| 201029 | 274671 | 298843 | 146709 | 154405 |
|Non-DataNode in same rack| 92687 | 182100 | 134704 | 277057 | 207532 |
|Non-DataNode in different rack| 245957 | 115076 | 203657 | 181819 | 116314 |
----
Below is the time taken by {{NetworkTopology.sortByDistance}} for a one GB file (eight blocks),
the values are in nanosecond.

*Without patch*

||Client on|| Run 1 || Run 2 || Run 3 || Run 4 || Run 5 ||
|Non-DataNode in same rack| 244535 | 282273 | 216524 | 4410825 | 339375 |

*With patch*

||Client on|| Run 1 || Run 2 || Run 3 || Run 4 || Run 5 ||
|Non-DataNode in same rack| 729701 | 5801405 | 613048 | 655345 | 506294 |


> getBlockLocations might not sort datanodes properly by distance
> ---------------------------------------------------------------
>
>                 Key: HDFS-10206
>                 URL: https://issues.apache.org/jira/browse/HDFS-10206
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Nandakumar
>         Attachments: HDFS-10206.000.patch
>
>
> If the DFSClient machine is not a datanode, but it shares its rack with some datanodes
of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}} might not put the local-rack
datanodes at the beginning of the sorted list. That is because the function didn't call {{networktopology.add(client);}}
to properly set the node's parent node; something required by {{networktopology.sortByDistance}}
to compute distance between two nodes in the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only distinguishes local
rack from remote rack, but it doesn't support general distance calculation to tell how remote
the rack is.
> {noformat}
> NetworkTopology.java
>   protected int getWeight(Node reader, Node node) {
>     // 0 is local, 1 is same rack, 2 is off rack
>     // Start off by initializing to off rack
>     int weight = 2;
>     if (reader != null) {
>       if (reader.equals(node)) {
>         weight = 0;
>       } else if (isOnSameRack(reader, node)) {
>         weight = 1;
>       }
>     }
>     return weight;
>   }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to address another
issue. Regardless of where we do the sorting, we still need fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object used by
DatanodeManager and requires Nodes stored in the topology to be {{DatanodeDescriptor}} for
block placement. So we need to make sure we don't pollute the  NetworkTopology if we plan
to fix it on the server side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message