accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2036) Mappers are not running locally
Date Fri, 03 Jan 2014 04:55:51 GMT


Josh Elser commented on ACCUMULO-2036:

Found enough code through the TaskAttemptImpl and MapTaskAttemptImpl, the StateMachineFactory
(and its transitions) and the RMContainerAllocator that make me confident that things are
still happening how we thought they are.

This was probably a huge waste of time for something we could have been relatively certain
about through testing, but oh well :P

> Mappers are not running locally
> -------------------------------
>                 Key: ACCUMULO-2036
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>            Reporter: Keith Turner
>            Assignee: Josh Elser
>             Fix For: 1.6.0
> I ran listscans in the Accumulo shell while running continuous verify on small cluster
and almost not map task were running locally.
> I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT.  Before that change
Accumulo would always store IP addrs.  Code like the following in o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits()
would translate IPs to hostnames.  
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}
> In my case I configured Accumulo to use hostnames, but not fully qualified ones.  So
I think the above code was just passing the non-quallified hostname through.   I suspected
hadoop wanted FQDN and changed the code to the following and mappers ran locally.  I need
to confirm what hadoop is expecting.  I think the above code will result in a FQDN if given
an IP,  so this is not an issue for 1.4 or 1.5.
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getCanonicalHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}

This message was sent by Atlassian JIRA

View raw message