hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-10208) Addendum for HDFS-9579: to handle the case when client machine can't resolve network path
Date Fri, 01 Apr 2016 16:37:25 GMT

     [ https://issues.apache.org/jira/browse/HDFS-10208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ming Ma updated HDFS-10208:
    Attachment: HDFS-10208-3.patch

Thanks [~brahmareddy]. I have updated the property description in core-default.xml.

>From the investigation of HDFS-10206, the current way of using Topology tree to compute
node distance by reference seems too expensive.  It means it needs to add nodes to the Topology
tree first which could become slow as the tree grows. It also means the tree size could grow
unbounded. To solve this issue, we can use network path string comparison instead without
Topology tree and the extra HashMap in ClientContext.

So in summary, the patch has three improvements:

* Handle the case the client can't resolve network path properly.
* Make the client-side topology resolution optional.
* Use string based comparison for network distance calculation.

> Addendum for HDFS-9579: to handle the case when client machine can't resolve network
> -----------------------------------------------------------------------------------------
>                 Key: HDFS-10208
>                 URL: https://issues.apache.org/jira/browse/HDFS-10208
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: HDFS-10208-2.patch, HDFS-10208-3.patch, HDFS-10208.patch
> If DFSClient runs on a machine that can't resolve network path, e.g.{{dnsToSwitchMapping.resolve}}
returns null, that will cause exception when it tries to create {{clientNode}}. In such case,
there is no need to create {{clientNode}} as null {{clientNode}} means its network distance
with any datanode is Integer.MAX_VALUE, which is what we want.
> {noformat}
>     clientNode = new NodeBase(clientHostName,
>         dnsToSwitchMapping.resolve(nodes).get(0));
> {noformat}

This message was sent by Atlassian JIRA

View raw message