accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2036) Mappers are not running locally
Date Mon, 06 Jan 2014 20:33:51 GMT


Josh Elser commented on ACCUMULO-2036:

bq. thanks for looking into this. Nice that you made the change for mapred and mapreduce.
The duplicated code drives me a bit nuts.

No problemo -- I agree on the duplicity. It's not ideal right now, but incremental improvements,
and whatnot.

bq. It does not seems there is a way to test this automatically. I am thinking we should create
a list of things that need to be manually verified before release and add this to it. I will
open a ticket for that.

Agreed. I sat and spun my wheels for a while trying to come up with some way to test this,
but couldn't come up with anything good (so I settled for manual verification). MapReduce/Yarn
might have some testing cluster/utility (ala minidfscluster) that might give us hints at a
nice, clean way to test this. That might be a good place to start looking.

> Mappers are not running locally
> -------------------------------
>                 Key: ACCUMULO-2036
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>            Reporter: Keith Turner
>            Assignee: Josh Elser
>              Labels: 16_qa_bug
>             Fix For: 1.6.0
> I ran listscans in the Accumulo shell while running continuous verify on small cluster
and almost not map task were running locally.
> I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT.  Before that change
Accumulo would always store IP addrs.  Code like the following in o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits()
would translate IPs to hostnames.  
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}
> In my case I configured Accumulo to use hostnames, but not fully qualified ones.  So
I think the above code was just passing the non-quallified hostname through.   I suspected
hadoop wanted FQDN and changed the code to the following and mappers ran locally.  I need
to confirm what hadoop is expecting.  I think the above code will result in a FQDN if given
an IP,  so this is not an issue for 1.4 or 1.5.
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getCanonicalHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}

This message was sent by Atlassian JIRA

View raw message