hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3234) Locality scheduling broken due to mismatch between IPs and hosts
Date Thu, 20 Oct 2011 17:28:10 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Hitesh Shah updated MAPREDUCE-3234:

    Attachment: MR-3234.1.patch

Adding a patch for some imports that should have been removed when the code to convert splitLocation
hostnames to ips was removed. 
> Locality scheduling broken due to mismatch between IPs and hosts
> ----------------------------------------------------------------
>                 Key: MAPREDUCE-3234
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3234
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Priority: Blocker
>         Attachments: MR-3234.1.patch
> I noticed that, on a single-rack cluster, I wasn't getting hardly any data locality.
The issue appears to be the code in RMContainerAllocator which changes the resource requests
to use IP addresses instead of hostnames:
> {code}
>            //host comes from data splitLocations which are hostnames. Containers
>            // use IP addresses.
>            //TODO Temporary fix for locality. Use resolvers from h-common. 
>            // Cache to make this more efficient ?
> {code}
> However, at least on my cluster, the resource manager sees node resources as hostnames,
not IPs. Removing this code fixed data locality.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message