hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rajeshbabu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
Date Sun, 28 Oct 2012 11:31:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485607#comment-13485607
] 

rajeshbabu commented on HBASE-6060:
-----------------------------------

@Jimmy,
Patch looks good and it properly reassigns the regions transitioning(which are in PENDING_OPEN,OPENING
state) to dead server by TM and also avoids double assignments.
But the regions which are in OPEN state may not be assigned. 
{code}
          RegionState rit = services.getAssignmentManager()
            .getRegionStates().getRegionTransitionState(e.getKey());
          if (processDeadRegion(e.getKey(), e.getValue(),
              this.services.getAssignmentManager(),
              this.server.getCatalogTracker())) {
            ServerName addressFromAM = this.services.getAssignmentManager()
              .getRegionStates().getRegionServerOfRegion(e.getKey());
            if (rit != null && !rit.isClosing() && !rit.isPendingClose() &&
!rit.isSplitting()) {
              // Skip regions that were in transition unless CLOSING or
              // PENDING_CLOSE
              LOG.info("Skip assigning region " + rit.toString());
{code}

If the region state is OPEN and server in RIT same as the dead server we can add it to toAssignRegions
to assign.

-> Before forcing region state to offline we are checking current state and if its other
than CLOSED or OFFLINE(forceNewPlan is true in case of assign call from TM) we calling unassing
on the region.
if the server in the RIT is dead we are deleting znode if its in M_ZK_REGION_CLOSING or RS_ZK_REGION_CLOSED
but in case of PENDING_OPEN,OPENING,OPEN case znode state wont be both. May be we can avoid
two network calls to zookeeper here by having region state check.
{code}
    if (!serverManager.isServerOnline(server)) {
      // delete the node. if no node exists need not bother.
      deleteClosingOrClosedNode(region);
      regionOffline(region);
      return;
    }
{code}
Please correct me if wrong. Great work Jimmy.
                
> Regions's in OPENING state from failed regionservers takes a long time to recover
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-6060
>                 URL: https://issues.apache.org/jira/browse/HBASE-6060
>             Project: HBase
>          Issue Type: Bug
>          Components: master, regionserver
>            Reporter: Enis Soztutar
>            Assignee: rajeshbabu
>             Fix For: 0.92.3, 0.94.3, 0.96.0
>
>         Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch,
6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch,
6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch,
6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch,
HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch,
trunk-6060.patch, trunk-6060_v2.patch
>
>
> we have seen a pattern in tests, that the regions are stuck in OPENING state for a very
long time when the region server who is opening the region fails. My understanding of the
process: 
>  
>  - master calls rs to open the region. If rs is offline, a new plan is generated (a new
rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE).
See HRegionServer.openRegion(), HMaster.assign()
>  - RegionServer, starts opening a region, changes the state in znode. But that znode
is not ephemeral. (see ZkAssign)
>  - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
>  - rs then opens the region, and changes znode from OPENING to OPENED
>  - when rs is killed between OPENING and OPENED states, then zk shows OPENING state,
and the master just waits for rs to change the region state, but since rs is down, that wont
happen. 
>  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these
kind of conditions. It periodically checks (every 10 sec by default) the regions in transition
to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout
is 30 min, which explains what you and I are seeing. 
>  - ServerShutdownHandler in Master does not reassign regions in OPENING state, although
it handles other states. 
> Lowering that threshold from the configuration is one option, but still I think we can
do better. 
> Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message