hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way
Date Tue, 07 Nov 2017 07:20:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241608#comment-16241608

Weiwei Yang commented on HDFS-11419:

Hi [~vagarychen]

Thank you for the response.

bq. NN will spend lot of time on trying to find available SSDs among the 500 DNs

Correct. Due to ALL_SSD policy, all 3 replicas need to be stored in SSD storage, only falls
back to DISK when no SSD volume available. From what I saw, this fall back took a long time,
and before when, all NN handlers were running {{BlockPlacementPolicyDefault.chooseDataNode}}
(per nn jstack dump). It looks like current mechanism to choose random nodes was too costy.

bq. tracks the available space of different storage types

Yes, if nodes can be picked up with volume space awareness, it can fix the problem. 

Appreciate your comments.

> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> ----------------------------------------------------------------------
>                 Key: HDFS-11419
>                 URL: https://issues.apache.org/jira/browse/HDFS-11419
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Chen Liang
>            Assignee: Chen Liang
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up calling into
{{chooseRandom}}, which will first find a random datanode by calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope, excludedNodes);{code}, then
it checks whether that returned datanode satisfies storage type requirement
> {code}storage = chooseStorage4Block(
>               chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, and runs
the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after a random
node is already returned.  We've seen a case where a cluster has a large number of datanodes,
while only a few satisfy the storage type condition. So, for the most part, this code blindly
picks random datanodes that do not satisfy the storage type requirement.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is that, given
a set of excluded nodes, it first finds a random datanodes, then if it is in excluded nodes
set, try find another random nodes. So the more excluded nodes there are, the more likely
a random node will be in the excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way that datanodes
can be found more efficiently. There are currently two different high level solutions we are
> 1. add some field to Node base types to describe the storage type info, and when searching
for a node, we take into account such field(s), and do not return node that does not meet
the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types, e.g. for one storage
type, there is one tree subset that connects all the nodes with that type. And one search
happens on only one such subset. So unexpected storage types are simply not in the search
> Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for pointing out
a wrong statement (corrected now) in the description. Any further comments are more than welcome.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message