hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
Date Wed, 12 Apr 2017 20:58:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966610#comment-15966610
] 

Arpit Agarwal commented on HDFS-11530:
--------------------------------------

Thanks for the great work [~linyiqun] and [~vagarychen].

I think we need some more stress testing/validation of the new network topology implementation
before we make it the default. Here's my suggestion:
{code}
    if (DFSConfigKeys.DFS_BLOCK_REPLICATOR_CLASSNAME_DEFAULT.getName()
        .equals(conf.get(DFSConfigKeys.DFS_BLOCK_REPLICATOR_CLASSNAME_KEY))) {
      networktopology = DFSNetworkTopology.getInstance(conf);
    } else {
      networktopology = NetworkTopology.getInstance(conf);
    }
{code}

Instead of using the new network topology whenever the BlockPlacementPolicy is {{BlockPlacementPolicyDefault}},
let's add a new configuration setting that allows choosing the NetworkTopology class. The
rest of the changes in this patch can go in while we continue testing the new topology implementation.
At some point in the future we can change the default via configuration.

> Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-11530
>                 URL: https://issues.apache.org/jira/browse/HDFS-11530
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, HDFS-11530.003.patch,
HDFS-11530.004.patch, HDFS-11530.005.patch, HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch,
HDFS-11530.009.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. But this
method is contained in new topology {{DFSNetworkTopology}} which is specified for HDFS. We
should update this and let {{BlockPlacementPolicyDefault}} use the new way since the original
way is inefficient.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message