hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Error reading task output
Date Tue, 21 Apr 2009 15:44:42 GMT
Aaron Kimball wrote:
> Cam,
> This isn't Hadoop-specific, it's how Linux treats its network configuration.
> If you look at /etc/host.conf, you'll probably see a line that says "order
> hosts, bind" -- this is telling Linux's DNS resolution library to first read
> your /etc/hosts file, then check an external DNS server.
> You could probably disable local hostfile checking, but that means that
> every time a program on your system queries the authoritative hostname for
> "localhost", it'll go out to the network. You'll probably see a big
> performance hit. The better solution, I think, is to get your nodes'
> /etc/hosts files squared away.

I agree

> You only need to do so once :)

No, you need to detect whenever the Linux networking stack has decided 
to add new entries to resolv.conf  or /etc/hosts and detect when they 
are inappropriate. Which is a tricky thing to do as there are some cases 
where you may actually be grateful that someone in the debian codebase 
decided that adding the local hostname as is actually a 
feature. I ended up writing a new SmartFrog component that can be 
configured to fail to start if the network is a mess, which is something 
worth pushing out.

as part of hadoop diagnostics, this test would be one of the things to 
deal with and at least warn on. "your hostname is local, you will not be 
visible over the network".


View raw message