hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kannan Muthukkaruppan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2174) Review how we handle addresses in HBase
Date Fri, 29 Jan 2010 23:41:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806562#action_12806562
] 

Kannan Muthukkaruppan commented on HBASE-2174:
----------------------------------------------

To fill in some more details, this is what we think happened during DNS flakiness:

A regionServer periodically sends a regionServerReport (RPC call) to the master. A HServerInfo
argument is passed as an argument and it identifies the sending region server's identity in
IP address format.

The master, in ServerManager class, maintains a serversToServerInfo map which is hostname
based. Every time a master receives a regionServerReport it converts the IP address based
name to a hostname via the info.getServerName() call. Normally this call returns the hostname,
but we suspect that during the DNS flakiness, it returned an IP address based string. And
so, this caused ServerManager.java to think that it was hearing from a new server. And this
lead to:

 HServerInfo storedInfo = serversToServerInfo.get(info.getServerName());
    if (storedInfo == null) {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Received report from unknown server -- telling it " +   <<============
          "to " + CALL_SERVER_STARTUP + ": " + info.getServerName());  <<============
      }

and bad things happened down the road (such as the region server registering itself multiple
times in Zookeeper, cluster coming down, etc.).

The above error message in our logs (example below) indeed identified the host in IP address
syntax, even though normally the getServerName call would return the info in hostname format.

2010-01-28 11:21:34,539 DEBUG org.apache.hadoop.hbase.master.ServerManager: Received report
from unknown server -- telling it to MSG_CALL_SERVER_STARTUP: 10.129.68.203,60020,1263605543210

--

Perhaps all we need to do is to change the ServerManager's internal maps to all be IP based?
That way we avoid/bypass the master having to look up the hostname on every heartbeat.


> Review how we handle addresses in HBase
> ---------------------------------------
>
>                 Key: HBASE-2174
>                 URL: https://issues.apache.org/jira/browse/HBASE-2174
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: Jean-Daniel Cryans
>             Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is
that addresses are handled differently in different parts of the code. We need to set a standard
and correct any inconsistencies.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message