hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Bowyer (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-3545) Possible liveness issue with MasterServerAddress in HRegionServer getMaster
Date Thu, 17 Feb 2011 17:33:24 GMT
Possible liveness issue with MasterServerAddress in HRegionServer getMaster 
----------------------------------------------------------------------------

                 Key: HBASE-3545
                 URL: https://issues.apache.org/jira/browse/HBASE-3545
             Project: HBase
          Issue Type: Bug
          Components: regionserver
    Affects Versions: 0.90.0
         Environment: 4 Node test cluster
2x Hbase master
3x Zookeeper nodes
4x RS
            Reporter: Greg Bowyer


As part of our evaluation of HBase we have been testing failure scenarios to see how HBase
fails in certain situations.

One of these is the outright failure of a HBase master.

What presently happens, if a HBase master is shutdown, is that the standby master becomes
the active master in the Zookeeper. At the same time the region servers fail to connect to
the dead master and typically fail their own heartbeats as part of the reportForDuty() method.

Following this the region server attempts to get a connection to a working HBase master, inside
the getMaster() method the first action is to get the address of a potentially working master
server from zookeeper. Following this the code is put into a tight loop whereupon it keeps
attempting to connect to the address of the master found in Zookeeper.

Unfortunately it appears that during master fail-over, it becomes possible to get the address
of the old, broken master, this address is then put into the connection attempt loop, whereupon
the region server attempts to infinitely connect to the failed, none existent master. At this
point nothing is able to break the loop in getMaster so the RS is unable to contact the master.

At the same time the new master is waiting patiently for the existing region servers as reported
in Zookeeper to re-establish contact with it.

Attached is a patch that rectifies this issue in our test cluster for both the 0.90.0 tag
and trunk versions (as of git SHA b72a24f71b67598e4077a9d1452f903082b0a9b7) of HBase.

This patch is also available in a forked repository here https://github.com/GregBowyer/hbase/commit/543f5903731ef6bbfd58c990e04a2c635e5c94b4

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message