hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (Reopened) (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (MAPREDUCE-3234) Locality scheduling broken due to mismatch between IPs and hosts
Date Sat, 22 Oct 2011 06:20:32 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vinod Kumar Vavilapalli reopened MAPREDUCE-3234:

> Locality scheduling broken due to mismatch between IPs and hosts
> ----------------------------------------------------------------
>                 Key: MAPREDUCE-3234
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3234
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Priority: Blocker
>         Attachments: MR-3234.1.patch
> I noticed that, on a single-rack cluster, I wasn't getting hardly any data locality.
The issue appears to be the code in RMContainerAllocator which changes the resource requests
to use IP addresses instead of hostnames:
> {code}
>            //host comes from data splitLocations which are hostnames. Containers
>            // use IP addresses.
>            //TODO Temporary fix for locality. Use resolvers from h-common. 
>            // Cache to make this more efficient ?
> {code}
> However, at least on my cluster, the resource manager sees node resources as hostnames,
not IPs. Removing this code fixed data locality.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message