accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2036) Mappers are not running locally
Date Fri, 03 Jan 2014 03:42:50 GMT


Josh Elser commented on ACCUMULO-2036:

Looks like you were correct about fqdn, Keith. From hadoop-2.2.0, this gets called when the
string looks like an IP, and the string is added otherwise.

  protected String resolveHost(String src) {
    String result = src; // Fallback in case of failure.
    try {
      InetAddress addr = InetAddress.getByName(src);
      result = addr.getHostName();
    } catch (UnknownHostException e) {
      LOG.warn("Failed to resolve address: " + src
          + ". Continuing to use the same.");
    return result;

Ok, dug down into the Java source and ultimately some man pages, but I found the information
I want:

The domain name queries carried out by gethostbyname() and gethostbyaddr() use a combination
of any or all of the name server named(8),  a  broken  out in /etc/host.conf.  The default
action is to query named(8), followed by /etc/hosts.

I think that explains when/how Java will be returning the FQDN from the alias.

Just need to find again where Hadoop is assigning the mappers to TaskTrackers (or rather Map
task to Container on NodeManager) to get the locality.

> Mappers are not running locally
> -------------------------------
>                 Key: ACCUMULO-2036
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>            Reporter: Keith Turner
>            Assignee: Josh Elser
>             Fix For: 1.6.0
> I ran listscans in the Accumulo shell while running continuous verify on small cluster
and almost not map task were running locally.
> I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT.  Before that change
Accumulo would always store IP addrs.  Code like the following in o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits()
would translate IPs to hostnames.  
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}
> In my case I configured Accumulo to use hostnames, but not fully qualified ones.  So
I think the above code was just passing the non-quallified hostname through.   I suspected
hadoop wanted FQDN and changed the code to the following and mappers ran locally.  I need
to confirm what hadoop is expecting.  I think the above code will result in a FQDN if given
an IP,  so this is not an issue for 1.4 or 1.5.
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getCanonicalHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}

This message was sent by Atlassian JIRA

View raw message