hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
Date Mon, 21 May 2012 18:55:41 GMT
Enis Soztutar created HBASE-6060:
------------------------------------

             Summary: Regions's in OPENING state from failed regionservers takes a long time
to recover
                 Key: HBASE-6060
                 URL: https://issues.apache.org/jira/browse/HBASE-6060
             Project: HBase
          Issue Type: Bug
          Components: master, regionserver
            Reporter: Enis Soztutar
            Assignee: Enis Soztutar


we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long
time when the region server who is opening the region fails. My understanding of the process:

 
 - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs
is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE).
See HRegionServer.openRegion(), HMaster.assign()
 - RegionServer, starts opening a region, changes the state in znode. But that znode is not
ephemeral. (see ZkAssign)
 - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
 - rs then opens the region, and changes znode from OPENING to OPENED
 - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the
master just waits for rs to change the region state, but since rs is down, that wont happen.

 - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind
of conditions. It periodically checks (every 10 sec by default) the regions in transition
to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout
is 30 min, which explains what you and I are seeing. 
 - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it
handles other states. 

Lowering that threshold from the configuration is one option, but still I think we can do
better. 

Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message