hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9150) Unnecessary DNS resolution attempts for logical URIs
Date Wed, 16 Jan 2013 16:56:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555198#comment-13555198

Daryn Sharp commented on HADOOP-9150:

Skimming the patch, to reduce adding more methods and complexity, maybe we should consider
either of the following:
# Default impl of {{getCanonicalUri()}} just returns {{getUri()}}.  Filesystem like DFS can
specifically override {{getCanonicalUri}} to call {{NetUtils.getCanonicalUri}}.  The advantage
is that it won't preclude other/future logical filesystems from utilizing a port.
# {{getCanonicalUri()}} continues to call {{NetUtils.getCanonicalUri}}.  Logical filesystems
should have a default port of -1 (ie. URI considers this as no port), so perhaps {{NetUtils.getCanonicalUri}}
can just return the given uri if there's no default port.

I lean towards #1.
> Unnecessary DNS resolution attempts for logical URIs
> ----------------------------------------------------
>                 Key: HADOOP-9150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9150
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3, ha, performance, viewfs
>    Affects Versions: 3.0.0, 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-9150.txt, hadoop-9150.txt, hadoop-9150.txt, hadoop-9150.txt,
log.txt, tracing-resolver.tgz
> In the FileSystem code, we accidentally try to DNS-resolve the logical name before it
is converted to an actual domain name. In some DNS setups, this can cause a big slowdown -
eg in one misconfigured cluster we saw a 2-3x drop in terasort throughput, since every task
wasted a lot of time waiting for slow "not found" responses from DNS.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message