hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8801) Region level read/write degradation
Date Tue, 25 Jun 2013 02:40:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692696#comment-13692696

Chao Shi commented on HBASE-8801:

Here is our proposed solutions (this is done by [~xingren23] [~boneylw] and me). We are looking
forward for advice from you experts.

h4. a) client-side blacklist
A very naive solution is to introduce znodes: /hbase/unhealth/<region>. Once a region
is set to read-only or write-only, we write something to the znode. Clients watch on /hbase/unhealth.
Whenever get notified, the client start to reject read or write requests accordingly.

This change should be simple as it does not touch anything at RS side. One shorthand is that
it does not handle split easily: when a region splits, the child regions lose the blacklist

[~boneylw] has made a prototype for approach a).

h4. b) server-side blacklist
Doing that at server-side is slight more complicated. We need to introduce a field in HRegionInfo
in META table, indicating whether a region is allowed for read/write.

1. A user calls RS to set a region’s read/write-only status.
2. The RS then updates the META table as well as its memory data structure. 
3. From now on, new read or write requests to that region will be rejected with an exception
(a subclass of NotServingRegionException). This could be done in HRegion#startRegionOperation,
similar to HBASE-7006 does.
4. When the client receives NotServingRegionException, it rejects requests at client side.
The current implementation talks to META table every retry. We need to cache the location
for a short time (e.g. ~10s to 1min), preventing touching META table too frequently.

On split, child regions inherit the blacklist settings.

If we get timeout exception between 2 and 3, we don’t know what HRegionInfo in META will
be. It is possible to get exception here but the mutation eventually get in. The risk is that
RS think of the region is inconsistent with META table. We think it is okay because it is
a tool and supposed to be executed manually. (Alternatively, we could abort RS just as split
transaction does, but it is an over-kill).

We have thought of using ZK rather than META table. We prefer META table as it is a central
place for storing region states.
> Region level read/write degradation
> -----------------------------------
>                 Key: HBASE-8801
>                 URL: https://issues.apache.org/jira/browse/HBASE-8801
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Chao Shi
> We would like to propose a tool for HBase administrators to disable read and/or write
for single region temporarily.
> Our HBase table at weibo.com is accessed by many clients. We have experienced the following
outage: due to bugs of user code, a small number of regions are reading at very high rate.
Requests to other regions at the same RSs are affected. In such scenario, the admin would
like to temporarily set the region to write-only (only forbids reads so writes can still being

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message