hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9150) Unnecessary DNS resolution attempts for logical URIs
Date Wed, 16 Jan 2013 23:36:13 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HADOOP-9150:
--------------------------------

    Attachment: hadoop-9150.txt

Attached patch implements something like what's described above.

Note that this changes behavior for those who are extending the FileSystem interface. It's
marked as Public and Stable, so we should probably add a release note for this. I think we
should allow it nonetheless -- it's not clear that the {{Stable}} marking there refers to
"stable for FileSystem developers to inherit from" vs "stable for developers to code against".
Since the changes are only to protected methods, we've only changed something for the former
and not the latter.
                
> Unnecessary DNS resolution attempts for logical URIs
> ----------------------------------------------------
>
>                 Key: HADOOP-9150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9150
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3, ha, performance, viewfs
>    Affects Versions: 3.0.0, 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-9150.txt, hadoop-9150.txt, hadoop-9150.txt, hadoop-9150.txt,
hadoop-9150.txt, log.txt, tracing-resolver.tgz
>
>
> In the FileSystem code, we accidentally try to DNS-resolve the logical name before it
is converted to an actual domain name. In some DNS setups, this can cause a big slowdown -
eg in one misconfigured cluster we saw a 2-3x drop in terasort throughput, since every task
wasted a lot of time waiting for slow "not found" responses from DNS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message