hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
Date Fri, 26 Oct 2012 21:27:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485209#comment-13485209

Jimmy Xiang commented on HBASE-6060:

@Stack, here is my understanding on the problem. Master calls a rs to open a region. Now,
in master memory, the region is in pending_open state with this rs' server name. Now the rs
dies.  When SSH starts, it goes to meta to find all the regions on this rs, minus those regions
already in transition, then assign the remaining regions. If the pending_open region (it could
be opening too depending on timing) was on this region server before, SSH will take care of
it.  Otherwise, if it was on a different region server, SSH will not pick it up.  In this
patch, I just times out the region transition so that tm can change the state and re-assign
it, instead of waiting for a long time (now, 20 minutes by default).

I'd like to make sure the region states in master memory is reliable.  Otherwise, it is of
not much use. So I think master always has region control.
In 0.96, I think region states is very reliable now. Of course, there could be bugs I am not
aware of yet.

@Ted, good point.  I will include the test. For EnvironmentEdgeManager, I will leave it to
another jira.

> Regions's in OPENING state from failed regionservers takes a long time to recover
> ---------------------------------------------------------------------------------
>                 Key: HBASE-6060
>                 URL: https://issues.apache.org/jira/browse/HBASE-6060
>             Project: HBase
>          Issue Type: Bug
>          Components: master, regionserver
>            Reporter: Enis Soztutar
>            Assignee: rajeshbabu
>             Fix For: 0.92.3, 0.94.3, 0.96.0
>         Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch,
6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch,
6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch,
6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch,
HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch,
> we have seen a pattern in tests, that the regions are stuck in OPENING state for a very
long time when the region server who is opening the region fails. My understanding of the
>  - master calls rs to open the region. If rs is offline, a new plan is generated (a new
rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE).
See HRegionServer.openRegion(), HMaster.assign()
>  - RegionServer, starts opening a region, changes the state in znode. But that znode
is not ephemeral. (see ZkAssign)
>  - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
>  - rs then opens the region, and changes znode from OPENING to OPENED
>  - when rs is killed between OPENING and OPENED states, then zk shows OPENING state,
and the master just waits for rs to change the region state, but since rs is down, that wont
>  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these
kind of conditions. It periodically checks (every 10 sec by default) the regions in transition
to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout
is 30 min, which explains what you and I are seeing. 
>  - ServerShutdownHandler in Master does not reassign regions in OPENING state, although
it handles other states. 
> Lowering that threshold from the configuration is one option, but still I think we can
do better. 
> Will investigate more. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message