hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9150) Unnecessary DNS resolution attempts for logical URIs
Date Wed, 16 Jan 2013 23:26:14 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555597#comment-13555597

Todd Lipcon commented on HADOOP-9150:

Hey Daryn.

I'm looking at implementing your #1 option above. Another issue, though, is that checkPath()
hard-codes {{NetUtils.getCanonicalUri}}. I don't really want to have to make all of HDFS,HFTP,HSFTP,WebHDFS
re-implement that code.

How does the following sound?
- Add {{protected FileSystem.canonicalizeUri(uri)}}. Default implementation would add getDefaultPort()
if the given URI has no port set, and if getDefaultPort() > 0
- Make the default implementation of {{FileSystem.getCanonicalUri()}} call {{canonicalizeUri(getUri())}}
- Change {{checkPath}} to call {{canonicalizeUri}}
- In all of the FileSystems which use real hostnames as their authority, override {{canoncalizeUri}}
to call {{NetUtils.canonicalizeUri}}
> Unnecessary DNS resolution attempts for logical URIs
> ----------------------------------------------------
>                 Key: HADOOP-9150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9150
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3, ha, performance, viewfs
>    Affects Versions: 3.0.0, 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-9150.txt, hadoop-9150.txt, hadoop-9150.txt, hadoop-9150.txt,
log.txt, tracing-resolver.tgz
> In the FileSystem code, we accidentally try to DNS-resolve the logical name before it
is converted to an actual domain name. In some DNS setups, this can cause a big slowdown -
eg in one misconfigured cluster we saw a 2-3x drop in terasort throughput, since every task
wasted a lot of time waiting for slow "not found" responses from DNS.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message