hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
Date Sun, 10 Jun 2012 04:42:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292470#comment-13292470
] 

stack commented on HBASE-6060:
------------------------------

bq. We need to consider regions in OPEN state also(along with OPENING) for assignment.

Yes.  Makes sense.

bq. you mean to say skip assignment of the regions in OFFLINE or PENDING_OPEN while processing
server shutdown, because they will be assigned as part of retry?

Yes.

bq. Lets suppose Region server went down after spawning OpenRegionHandler and before transitioning
to OPENING then its SSH responsibility to assign regions in OFFLINE/PENDING_OPEN.

This is a good one.  I'd argue this is a hole in our transition states.  Region is OFFLINE
but the regionserver is supposed to have control.  I'd suggest that a regionserver not return
from open region until its moved the znode from PENDING_OPEN to OPENING (need to look more
into what would be involved making this happen -- and what to do on bulk assign though it
seems we drop a bunch of the handling already when bulk assigning).  Shouldn't region belong
to master or regionserver without a gray area in-between while PENDING_OPEN is going on?

bq. during opening region if destination server details not updated in META then the region
wont present in hris...

This is also a good one.  I found it earlier this afternoon making your tests work w/ the
alternate patch (There is another 'hole' that I found in the case where there are no regions
on the server that died; in this case we'll skip out early and not find associated OPENING
or OPEN regions -- my patch is much larger than four lines now (smile)).

What do you think of this alternate solution Rajeshbabu?  Assuming I close the above holes,
do you think it easier to reason about?  We don't have to keep up state distributed across
AM and SSH (I think it more clear whats going on), we don't have to use a special RegionPlan
to signal the single region assign method to give up on its assign (I'd think we'd want regions
assigned by the single-assign call rather than the bulk assign since the single assign does
more checking), and we are not built atop the current state of regionplans, which seems a
little bit of an indirect foundation given we are trying to keep up RegionState in regionsInTransition.

Thanks for the thorough review.
                
> Regions's in OPENING state from failed regionservers takes a long time to recover
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-6060
>                 URL: https://issues.apache.org/jira/browse/HBASE-6060
>             Project: HBase
>          Issue Type: Bug
>          Components: master, regionserver
>            Reporter: Enis Soztutar
>            Assignee: rajeshbabu
>             Fix For: 0.96.0, 0.94.1, 0.92.3
>
>         Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch,
6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060_alternative_suggestion.txt,
6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, HBASE-6060-92.patch,
HBASE-6060-94.patch
>
>
> we have seen a pattern in tests, that the regions are stuck in OPENING state for a very
long time when the region server who is opening the region fails. My understanding of the
process: 
>  
>  - master calls rs to open the region. If rs is offline, a new plan is generated (a new
rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE).
See HRegionServer.openRegion(), HMaster.assign()
>  - RegionServer, starts opening a region, changes the state in znode. But that znode
is not ephemeral. (see ZkAssign)
>  - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
>  - rs then opens the region, and changes znode from OPENING to OPENED
>  - when rs is killed between OPENING and OPENED states, then zk shows OPENING state,
and the master just waits for rs to change the region state, but since rs is down, that wont
happen. 
>  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these
kind of conditions. It periodically checks (every 10 sec by default) the regions in transition
to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout
is 30 min, which explains what you and I are seeing. 
>  - ServerShutdownHandler in Master does not reassign regions in OPENING state, although
it handles other states. 
> Lowering that threshold from the configuration is one option, but still I think we can
do better. 
> Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message