hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajith S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11104) BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance
Date Sat, 12 Nov 2016 14:16:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15659720#comment-15659720
] 

Ajith S commented on HDFS-11104:
--------------------------------

Would like to work on this, can u please assign to me

> BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-11104
>                 URL: https://issues.apache.org/jira/browse/HDFS-11104
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Doris Gu
>
> if client transfer favoredNodes when it writes files into hdfs,chooseTarget in BlockPlacementPolicyDefault
prior chooseTarget in turn:
> {quote}
>     DatanodeStorageInfo[] chooseTarget(String src,
>       int numOfReplicas,
>       Node writer,
>       Set<Node> excludedNodes,
>       long blocksize,
>       List<DatanodeDescriptor> favoredNodes,
>       BlockStoragePolicy storagePolicy) {
>     try {
>     ...
>    *for (int i = 0; i < favoredNodes.size() && results.size() < numOfReplicas;
i++)* {
>         DatanodeDescriptor favoredNode = favoredNodes.get(i);
>         // Choose a single node which is local to favoredNode.
>         // 'results' is updated within chooseLocalNode
>         final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
>             favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
>             results, avoidStaleNodes, storageTypes, false);
>       ...
> {quote}
> why not shuffle it here? Make block more balanced, save the cost balancer will pay and
make cluster more stable.
> {quote}
> for (DatanodeDescriptor favoredNode : DFSUtil.shuffle(favoredNodes.toArray(new DatanodeDescriptor[favoredNodes.size()])))

> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message