hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-19144) [RSgroups] Regions assigned to a RSGroup all go to FAILED_OPEN state when all servers in the group are unavailable
Date Tue, 31 Oct 2017 22:47:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227729#comment-16227729
] 

Andrew Purtell edited comment on HBASE-19144 at 10/31/17 10:46 PM:
-------------------------------------------------------------------

[~stack] reported the same thing with branch-2 on HBASE-17580.
 
Seems the decision we made to make FAILED_OPEN state something an operator has to resolve
by hand, or by hbck (someday), or by master failover, is a nasty operational wrinkle when
using RSgroups, because the possibility of all servers in a RSGroup going down is higher than
for the whole cluster.

I wonder if we can more generally consider kicking FAILED_OPEN state assignments whenever
a new server shows up, with hysteresis such that if a few servers show up in a short period
of time we don't pile assignments all on the first one to show up. 


was (Author: apurtell):
[~stack] reported the same thing with branch-2 on HBASE-17580.
 
Seems the decision we made to make FAILED_OPEN state something an operator has to resolve
by hand, or by hbck (someday), or by master failover, is a nasty operational wrinkle when
using RSgroups, because the possibility of all servers in a RSGroup going down is higher than
for the whole cluster.

> [RSgroups] Regions assigned to a RSGroup all go to FAILED_OPEN state when all servers
in the group are unavailable
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-19144
>                 URL: https://issues.apache.org/jira/browse/HBASE-19144
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 3.0.0, 1.4.0, 1.5.0
>            Reporter: Andrew Purtell
>
> After all servers in the RSgroup are down the regions cannot be opened anywhere and transition
rapidly into FAILED_OPEN state.
>  
> 2017-10-31 21:06:25,449 INFO [ProcedureExecutor-13] master.RegionStates: Transition {c6c8150c9f4b8df25ba32073f25a5143
state=OFFLINE, ts=1509483985448, server=node-5.cluster,16020,1509482700768} to {c6c8150c9f4b8df25ba32073f25a5143
state=FAILED_OPEN, ts=1509483985449, server=node-5.cluster,16020,1509482700768}
> 2017-10-31 21:06:25,449 WARN [ProcedureExecutor-13] master.RegionStates: Failed to open/close
d4e2f173e31ffad6aac140f4bd7b02bc on node-5.cluster,16020,1509482700768, set to FAILED_OPEN
>  
> Any region in FAILED_OPEN state has to be manually reassigned, or the master can be restarted
and this will also cause reattempt of assignment of any regions in FAILED_OPEN state. This
is not unexpected but is an operational headache. It would be better if the RSGroupInfoManager
could automatically kick reassignments of regions in FAILED_OPEN state when servers rejoin
the cluster. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message