Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 13 Sep 2016 05:30:21 +0000 (UTC)
From: "Phil Yang (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12996265.1470833597000.556267.1473744621634@Atlassian.JIRA>
In-Reply-To: <JIRA.12996265.1470833597000@Atlassian.JIRA>
References: <JIRA.12996265.1470833597000@Atlassian.JIRA> <JIRA.12996265.1470833597107@arcas>
Subject: [jira] [Comment Edited] (HBASE-16388) Prevent client threads being
 blocked by only one slow region server
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 13 Sep 2016 05:30:23 -0000


    [ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486310#comment-15486310 ] 

Phil Yang edited comment on HBASE-16388 at 9/13/16 5:30 AM:
------------------------------------------------------------

If a user connect to a cluster with N region servers, we expected that if one region server is slow and all requests to it timed out, at a higher level which can monitor the service using HBase client, there should be only 1/N requests failed/timeout. If the failed requests are more than 1/N, especially much more than 1/N, the limit should be change to a lower number. 

And if there is no slow server in the cluster, but clients throw SBE, there may be two reasons: the limit is too strict, or the regions are not balanced and much more than 1/N keys belong to the server.

So the proper limit should be several times of 1/N*threadNumber, too high may reduce the availability when there are slow servers, too low may prevent normal requests. And a relatively  balanced cluster is required.


was (Author: yangzhe1991):
If a user connect to a cluster with N region servers, we expected that if one region server is slow and all requests to it timed out, at a higher level which can monitor the service using HBase client, there should be only 1/N requests failed/timeout. If the failed requests are more than 1/N, especially much more than 1/N, the limit should be change to a lower number. 

And if there is no slow server in the cluster, but clients throw SBE, there may be two reasons: the limit is too strict, or the regions are not balanced and much more than 1/N keys belong to the server.

So the proper limit should be several times of 1/N*threadNumber, too high may reduce the availability when there are slow servers, too low may prevent normal requests.

> Prevent client threads being blocked by only one slow region server
> -------------------------------------------------------------------
>
>                 Key: HBASE-16388
>                 URL: https://issues.apache.org/jira/browse/HBASE-16388
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Phil Yang
>            Assignee: Phil Yang
>         Attachments: HBASE-16388-branch-1-v1.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch
>
>
> It is a general use case for HBase's users that they have several threads/handlers in their service, and each handler has its own Table/HTable instance. Generally users think each handler is independent and won't interact each other.
> However, in an extreme case, if a region server is very slow, every requests to this RS will timeout, handlers of users' service may be occupied by the long-waiting requests even requests belong to other RS will also be timeout.
> For example: 
> If we have 100 handlers in a client service(timeout is 1000ms) and HBase has 10 region servers whose average response time is 50ms. If no region server is slow, we can handle 2000 requests per second.
> Now this service's QPS is 1000. If there is one region server very slow and all requests to it will be timeout. Users hope that only 10% requests failed, and 90% requests' response time is still 50ms, because only 10% requests are located to the slow RS. However, each second we have 100 long-waiting requests which exactly occupies all 100 handles. So all handlers is blocked, the availability of this service is almost zero.
> To prevent this case, we can limit the max concurrent requests to one RS in process-level. Requests exceeding the limit will throws ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above case, if we set this limit to 20, only 20 handlers will be occupied and other 80 handlers can still handle requests to other RS. The availability of this service is 90% as expected.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)