hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15317) Improve NetworkTopology chooseRandom's loop
Date Sat, 17 Mar 2018 05:25:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403275#comment-16403275

Xiao Chen commented on HADOOP-15317:

Discussed with [~eddyxu] about this a little bit, uploaded [^HADOOP-15317.02.patch]

We have several options:
 # Similar to patch 1 to use a data structure to track the choose results so we can terminate
 # Use a ridiculous big number like (10 * numOfDatanodes) to guarantee termination
 # Instead of random choosing in every loop, we only randomize the initial index, and then
loop through every node exactly once
 # Random once to get a number N, and only choose the Nth valid node

#1 will result in potentially largest memory consumption, #2 is minimal change but could
be the most CPU-heavy since it's just adding a termination to the current while loop.

#3 and #4 should be the least resource-heavy. But #3 has the problem that the probability
of each node being chosen isn't evenly distributed. Specifically, the node immediately after
excluded node(s) will have a higher probability of being chosen. Patch went with #4, whose
only draw back is to make best case slower.

Perhaps we can have a threshold (e.g. when availableNodes > numInScopeNodes / 2, we go
with the simple stupid random loop in existing code, so #4's best case slowness is covered.

Also thanks for the review [~ajayydv],  comments addressed.

> Improve NetworkTopology chooseRandom's loop
> -------------------------------------------
>                 Key: HADOOP-15317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15317
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
>            Priority: Major
>         Attachments: HADOOP-15317.01.patch, HADOOP-15317.02.patch
> Recently we found a postmortem case where the ANN seems to be in an infinite loop. From
the logs it seems it just went through a rolling restart, and DNs are getting registered.
> Later the NN become unresponsive, and from the stacktrace it's inside a do-while loop
inside {{NetworkTopology#chooseRandom}} - part of what's done in HDFS-10320.
> Going through the code and logs I'm not able to come up with any theory (thought about
incorrect locking, or the Node object being modified outside of NetworkTopology, both seem
impossible) why this is happening, but we should eliminate this loop.
> stacktrace:
> {noformat}
>  Stack:
> java.util.HashMap.hash(HashMap.java:338)
> java.util.HashMap.containsKey(HashMap.java:595)
> java.util.HashSet.contains(HashSet.java:203)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:786)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:732)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:757)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:692)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:573)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:461)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:368)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:243)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:115)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4AdditionalDatanode(BlockManager.java:1596)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3599)
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:717)
> {noformat}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message