hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doris Gu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-11104) BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance
Date Fri, 04 Nov 2016 09:36:59 GMT
Doris Gu created HDFS-11104:
-------------------------------

             Summary: BlockPlacementPolicyDefault choose favoredNodes in turn which may cause
imbalance
                 Key: HDFS-11104
                 URL: https://issues.apache.org/jira/browse/HDFS-11104
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Doris Gu


if client transfer favoredNodes when it writes files into hdfs,chooseTarget in BlockPlacementPolicyDefault
prior chooseTarget in turn:
{quote}
    DatanodeStorageInfo[] chooseTarget(String src,
      int numOfReplicas,
      Node writer,
      Set<Node> excludedNodes,
      long blocksize,
      List<DatanodeDescriptor> favoredNodes,
      BlockStoragePolicy storagePolicy) {
    try {
    ...

   *for (int i = 0; i < favoredNodes.size() && results.size() < numOfReplicas;
i++)* {
        DatanodeDescriptor favoredNode = favoredNodes.get(i);
        // Choose a single node which is local to favoredNode.
        // 'results' is updated within chooseLocalNode
        final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
            favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
            results, avoidStaleNodes, storageTypes, false);
      ...
{quote}
why not shuffle it?
{quote}
 *for (DatanodeDescriptor favoredNode : DFSUtil.shuffle(favoredNodes.toArray(new DatanodeDescriptor[favoredNodes.size()])))
*
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org


Mime
View raw message