hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
Date Mon, 11 Jun 2012 05:42:43 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292646#comment-13292646
] 

ramkrishna.s.vasudevan commented on HBASE-6060:
-----------------------------------------------

@Stack
Few comments,
First of all we need to do some more changes if we want to make the above change in handleRegion
{code}
          // Should see OPENING after we have asked it to OPEN or additional
          // times after already being in state of OPENING
          if (regionState == null ||
              (!regionState.isPendingOpen() && !regionState.isOpening())) {
            LOG.warn("Received OPENING for region " +
                prettyPrintedRegionName +
                " from server " + data.getOrigin() + " but region was in " +
                " the state " + regionState + " and not " +
                "in expected PENDING_OPEN or OPENING states");
{code}
Second is, you mean like the above change of making pending_open after the RPC call along
with the changes in the earlier patches like having the new items lik 'ritintersection' and
'outstandingRegionPlans'?
Here there is one concern is, if before we update the state to PENDING_OPEN in master side
if the OpenRegionHandler has really transitioned the node to OPENING then the inmemory state
will also be changing to OPENING after we correct the code above i have mentioned.  Now in
that case the OPENING state will be rewritten to PENDING_OPEN? So we may need to add a check
to see if already a change has happened in the REgionState.


The problem with Rajesh's patch is (6060_suggestion_toassign_rs_wentdown_beforerequest.patch:)
{code}
+      if ((region.getState() == RegionState.State.OFFLINE)
+          && (region.getState() == RegionState.State.PENDING_OPEN)) {
+        regionPlans.remove(region.getRegion());
+      }
{code}
What if the RS went down befor sending RPC. The SSH collected the regionPlan but before he
could remove the collected regionplan in the above code, if the master completed the retry
assignment and the OpenRegionHandler has changed to OPENING and the state is now Opening,
then we will again try to assign thro SSH.

The above problem can happen in the change that you mentioned also like moving PENDING_OPEN
after RPC.

So can we take a copy of RIT befor forming the RegionPlan and work based on that?  Will update
the change that we are suggesting in some time.




                
> Regions's in OPENING state from failed regionservers takes a long time to recover
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-6060
>                 URL: https://issues.apache.org/jira/browse/HBASE-6060
>             Project: HBase
>          Issue Type: Bug
>          Components: master, regionserver
>            Reporter: Enis Soztutar
>            Assignee: rajeshbabu
>             Fix For: 0.96.0, 0.94.1, 0.92.3
>
>         Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch,
6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060_alternative_suggestion.txt,
6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch,
HBASE-6060-92.patch, HBASE-6060-94.patch
>
>
> we have seen a pattern in tests, that the regions are stuck in OPENING state for a very
long time when the region server who is opening the region fails. My understanding of the
process: 
>  
>  - master calls rs to open the region. If rs is offline, a new plan is generated (a new
rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE).
See HRegionServer.openRegion(), HMaster.assign()
>  - RegionServer, starts opening a region, changes the state in znode. But that znode
is not ephemeral. (see ZkAssign)
>  - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
>  - rs then opens the region, and changes znode from OPENING to OPENED
>  - when rs is killed between OPENING and OPENED states, then zk shows OPENING state,
and the master just waits for rs to change the region state, but since rs is down, that wont
happen. 
>  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these
kind of conditions. It periodically checks (every 10 sec by default) the regions in transition
to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout
is 30 min, which explains what you and I are seeing. 
>  - ServerShutdownHandler in Master does not reassign regions in OPENING state, although
it handles other states. 
> Lowering that threshold from the configuration is one option, but still I think we can
do better. 
> Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message