hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rural Hunter <ruralhun...@gmail.com>
Subject Why hadoop/hbase uses DNS/hosts/hostname in such a strange way?
Date Tue, 10 Dec 2013 01:31:45 GMT
Hi,

I have configured a hadoop/hbase cluster recently and found it's really 
a mess with all those DNS, hostname and /etc/hosts configuration. There 
are many questions related to this all over the internet. So I'm 
wondering why hadoop/hbase designed in such a strange way, which is very 
abnormal comparing with other network/distribution applications. In 
normal applications, DNS is used to indentify other servers(logical or 
physical), not the server itself. But I'm seeing this weired behavior in 
hadoop/hbase.

Say we have server1 and server2 configured this way:

server1(ip 192.168.1.2)
hostname: server1
/etc/hosts:
127.0.0.1    localhost,server1
192.168.1.3    server2

server2(ip 192.168.1.3)
hostname: server2
/etc/hosts:
127.0.0.1    localhost,server2
192.168.1.2    server1

With the configuration above, I'm seesing many cases hadoop/hbase trying 
to connect to localhost while it actaully should connect to another 
server. I believe this is because server1 reported its hostname as 
'localhost' to server2 and server2 tries to use 'localhost' to connect 
to server1. But it shouldn't work that way. In normal network 
applications, server2 shouldn't try to connect to server1 with what 
server1 reported. If server2 inits the connection, it should use the DNS 
or /etc/hosts to resolve server1. If server 1 inits the connection, 
server2 should use the ip it gets from the already established 
connection from server1. There shouldn't be any confusion or mess.

I don't see why hadoop/hbase can not use the same logic to handle the 
DNS/hosts/hostname mess. Anyone can resolve my confusion?

Mime
View raw message