Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 11 Apr 2017 21:16:41 +0000 (UTC)
From: "Chen Liang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13055893.1489486130000.260044.1491945401881@Atlassian.JIRA>
In-Reply-To: <JIRA.13055893.1489486130000@Atlassian.JIRA>
References: <JIRA.13055893.1489486130000@Atlassian.JIRA> <JIRA.13055893.1489486130156@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HDFS-11530) Use HDFS specific network topology to
 choose datanode in BlockPlacementPolicyDefault
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 11 Apr 2017 21:16:47 -0000


     [ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chen Liang updated HDFS-11530:
------------------------------
    Attachment: HDFS-11530.008.patch

Attaching v008 patch to fix the issue. [~linyiqun] [~arpitagarwal] would you please take a look on this one some time? As v008 patch made a few small changes but they kind of break some earlier assumptions.

More specifically, {{TestBlockManager.testSafeModeIBR}} fail seems to be caused by that the {{add(Node node)}} was called more than once on the same node, with different storage type info. This violates the earlier assumption that nodes only get called by add once, or at least always the same storage info. The reason seems that this tests is simulating datanode fails and this is why nodes get added twice. This seems to be a valid case so I added {{DFSTopologyNodeImpl.updateExistingDatanode}} for this.

On the other hand, {{TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed}} fail seems to be caused by that the {{excludedNodes}} set does not have to be {{DatanodeDescriptor}}, but can also be {{DatanodeInfo}} instead. (Although it's a bit suspicious to me why this is the only test that failed due to this). Assuming this is valid, I modified {{DFSNetworkTopology.chooseRandomWithStorageType}} to do some more check.

I do appreciate any comments. Basically I made the changes here assuming the tests are doing the right thing, but it might very well be that the tests themselves should be modified instead. 

> Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-11530
>                 URL: https://issues.apache.org/jira/browse/HDFS-11530
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. But this method is contained in new topology {{DFSNetworkTopology}} which is specified for HDFS. We should update this and let {{BlockPlacementPolicyDefault}} use the new way since the original way is inefficient.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org