Return-Path: <hdfs-issues-return-173440-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id E58D1200BB6
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Fri,  4 Nov 2016 10:37:00 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id E453F160B07; Fri,  4 Nov 2016 09:37:00 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 382E8160AE9
	for <archive-asf-public@cust-asf.ponee.io>; Fri,  4 Nov 2016 10:37:00 +0100 (CET)
Received: (qmail 94821 invoked by uid 500); 4 Nov 2016 09:36:59 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 94801 invoked by uid 99); 4 Nov 2016 09:36:59 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Nov 2016 09:36:59 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 4B1E62C1F56
	for <hdfs-issues@hadoop.apache.org>; Fri,  4 Nov 2016 09:36:59 +0000 (UTC)
Date: Fri, 4 Nov 2016 09:36:59 +0000 (UTC)
From: "Doris Gu (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13017996.1478252176000.186646.1478252219304@Atlassian.JIRA>
In-Reply-To: <JIRA.13017996.1478252176000@Atlassian.JIRA>
References: <JIRA.13017996.1478252176000@Atlassian.JIRA> <JIRA.13017996.1478252176760@arcas>
Subject: [jira] [Created] (HDFS-11104) BlockPlacementPolicyDefault choose
 favoredNodes in turn which may cause imbalance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Fri, 04 Nov 2016 09:37:01 -0000

Doris Gu created HDFS-11104:
-------------------------------

             Summary: BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance
                 Key: HDFS-11104
                 URL: https://issues.apache.org/jira/browse/HDFS-11104
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Doris Gu


if client transfer favoredNodes when it writes files into hdfs,chooseTarget in BlockPlacementPolicyDefault prior chooseTarget in turn:
{quote}
    DatanodeStorageInfo[] chooseTarget(String src,
      int numOfReplicas,
      Node writer,
      Set<Node> excludedNodes,
      long blocksize,
      List<DatanodeDescriptor> favoredNodes,
      BlockStoragePolicy storagePolicy) {
    try {
    ...

   *for (int i = 0; i < favoredNodes.size() && results.size() < numOfReplicas; i++)* {
        DatanodeDescriptor favoredNode = favoredNodes.get(i);
        // Choose a single node which is local to favoredNode.
        // 'results' is updated within chooseLocalNode
        final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
            favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
            results, avoidStaleNodes, storageTypes, false);
      ...
{quote}
why not shuffle it?
{quote}
 *for (DatanodeDescriptor favoredNode : DFSUtil.shuffle(favoredNodes.toArray(new DatanodeDescriptor[favoredNodes.size()]))) *
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org