hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eran Hirsch (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-5067) HMaster uses wrong name for address (in stand-alone mode)
Date Mon, 19 Dec 2011 18:49:31 GMT
HMaster uses wrong name for address (in stand-alone mode)

                 Key: HBASE-5067
                 URL: https://issues.apache.org/jira/browse/HBASE-5067
             Project: HBase
          Issue Type: Bug
          Components: master
    Affects Versions: 0.90.4
            Reporter: Eran Hirsch

When setting the configuration option "hbase.master.dns.interface" (and optional "hbase.master.dns.nameserver")
to non-default values,

it is EXPECTED that the master node would report its fully qualified dns name when registering
in ZooKeeper,
BUT INSTEAD, the machines hostname is taken instead.

For example, my machine is called (aka "its hostname is...") "machine1" but it's name in the
network is "machine1.our-dev-network.my-corp.com", so to find this machine's IP anywhere on
the network i would need to query for the whole name (because trying to find "machine1" is
ambiguous on a network).

Why is this a bug, because when trying to connect to this stand-alone hbase installation from
outside the machine it is running on, when querying ZK for /hbase/master we get only the "machine1"
part, and then fail with an unresolvable address for the master (which later even gives a
null pointer because of a missing null check).

This is the stack trace when calling HTable's c'tor:
java.lang.IllegalArgumentException: hostname can't be null
	at java.net.InetSocketAddress.<init>(InetSocketAddress.java:139) ~[na:1.7.0_02]
	at org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108) ~[hbase-0.90.4.jar:0.90.4]
	at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64) ~[hbase-0.90.4.jar:0.90.4]
	at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82)
	at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:579)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:688)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:688)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:173) ~[hbase-0.90.4.jar:0.90.4]
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147) ~[hbase-0.90.4.jar:0.90.4]


Why this happens?
1. When building the HMaster object we correctly use the static 'getMyAddress(conf)' to read
the configuration options, and then to try and resolve the machine's ip. This method returns
the full qualified name correctly, and this is then used to construct an 'HServerAddress'
object which is locally stored as 'a'.
2. So far so good, but now, instead of using this object as the value for the master's 'address'
field the code goes on to initialize the 'rpcServer' field. As part of this calls the static
'HBaseRPC.getServer' method is called with, among others, the HServerAddress's BIND ADDRESS
(aka the IP) that we have just built.
3. But now, when we finally get to setting the value for HMaster's 'address' field, we initialize
a NEW HServerAddress initialized with rpcServer.getListenerAddress() (which is basically the
IP we just gave it, with a new listening port.
4. HServerAddress calls 'getAddress().getHostName()' on this address object, which would return
the local hostname of the machine, because the IP would be resolved locally by the machine,
and not using a nameserver.

So eventually, the fully qualified name computed in step 1 is NOT USED in any way, instead,
all further processing is done on the IP address of the host (and its local resolving to the


What should happen?
The 'HMaster.address' field should be set to an address which is made of the fully qualified
name retrieved in step 1, combined with the port retrieved from the rpcServer computed at
step 2.



1. It seems that the 'HBaseServer' c'tor (which is called when 'HBaseRPC.getServer()' static
method is called) is faulty as it doesn't use the port number sent to it in effect (it sets
the local 'port' field to it, but then overrides it without ever reading it later on, with
the port returned from the new 'Listener' object. This might be a bug, but i have not checked
it enough.

2. The same bug with the master node could repeat itself in the region server code, but i
haven't checked that at all.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message