hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doris Gu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-11104) BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance
Date Fri, 04 Nov 2016 09:36:59 GMT
Doris Gu created HDFS-11104:

             Summary: BlockPlacementPolicyDefault choose favoredNodes in turn which may cause
                 Key: HDFS-11104
                 URL: https://issues.apache.org/jira/browse/HDFS-11104
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Doris Gu

if client transfer favoredNodes when it writes files into hdfs,chooseTarget in BlockPlacementPolicyDefault
prior chooseTarget in turn:
    DatanodeStorageInfo[] chooseTarget(String src,
      int numOfReplicas,
      Node writer,
      Set<Node> excludedNodes,
      long blocksize,
      List<DatanodeDescriptor> favoredNodes,
      BlockStoragePolicy storagePolicy) {
    try {

   *for (int i = 0; i < favoredNodes.size() && results.size() < numOfReplicas;
i++)* {
        DatanodeDescriptor favoredNode = favoredNodes.get(i);
        // Choose a single node which is local to favoredNode.
        // 'results' is updated within chooseLocalNode
        final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
            favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
            results, avoidStaleNodes, storageTypes, false);
why not shuffle it?
 *for (DatanodeDescriptor favoredNode : DFSUtil.shuffle(favoredNodes.toArray(new DatanodeDescriptor[favoredNodes.size()])))

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message