hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
Date Tue, 11 Apr 2017 21:16:41 GMT

     [ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chen Liang updated HDFS-11530:
    Attachment: HDFS-11530.008.patch

Attaching v008 patch to fix the issue. [~linyiqun] [~arpitagarwal] would you please take a
look on this one some time? As v008 patch made a few small changes but they kind of break
some earlier assumptions.

More specifically, {{TestBlockManager.testSafeModeIBR}} fail seems to be caused by that the
{{add(Node node)}} was called more than once on the same node, with different storage type
info. This violates the earlier assumption that nodes only get called by add once, or at least
always the same storage info. The reason seems that this tests is simulating datanode fails
and this is why nodes get added twice. This seems to be a valid case so I added {{DFSTopologyNodeImpl.updateExistingDatanode}}
for this.

On the other hand, {{TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed}} fail seems
to be caused by that the {{excludedNodes}} set does not have to be {{DatanodeDescriptor}},
but can also be {{DatanodeInfo}} instead. (Although it's a bit suspicious to me why this is
the only test that failed due to this). Assuming this is valid, I modified {{DFSNetworkTopology.chooseRandomWithStorageType}}
to do some more check.

I do appreciate any comments. Basically I made the changes here assuming the tests are doing
the right thing, but it might very well be that the tests themselves should be modified instead.

> Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
> ------------------------------------------------------------------------------------
>                 Key: HDFS-11530
>                 URL: https://issues.apache.org/jira/browse/HDFS-11530
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, HDFS-11530.003.patch,
HDFS-11530.004.patch, HDFS-11530.005.patch, HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. But this
method is contained in new topology {{DFSNetworkTopology}} which is specified for HDFS. We
should update this and let {{BlockPlacementPolicyDefault}} use the new way since the original
way is inefficient.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message