hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phil Yang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-16388) Prevent client threads being blocked by only one slow region server
Date Wed, 10 Aug 2016 12:53:20 GMT
Phil Yang created HBASE-16388:

             Summary: Prevent client threads being blocked by only one slow region server
                 Key: HBASE-16388
                 URL: https://issues.apache.org/jira/browse/HBASE-16388
             Project: HBase
          Issue Type: New Feature
            Reporter: Phil Yang
            Assignee: Phil Yang

It is a general use case for HBase's users that they have several threads/handlers in their
service, and each handler has its own Table/HTable instance. Generally users think each handler
is independent and won't interact each other.

However, in an extreme case, if a region server is very slow, every requests to this RS will
timeout, handlers of users' service may be occupied by the long-waiting requests even requests
belong to other RS will also be timeout.

For example: 
If we have 100 handlers in a client service(timeout is 1000ms) and HBase has 10 region servers
whose average response time is 50ms. If no region server is slow, we can handle 2000 requests
per second.
Now this service's QPS is 1000. If there is one region server very slow and all requests to
it will be timeout. Users hope that only 10% requests failed, and 90% requests' response time
is still 50ms, because only 10% requests are located to the slow RS. However, each second
we have 100 long-waiting requests which exactly occupies all 100 handles. So all handlers
is blocked, the availability of this service is almost zero.

To prevent this case, we can limit the max concurrent requests to one RS in process-level.
Requests exceeding the limit will throws ServerBusyException(extends DoNotRetryIOE) immediately
to users. In the above case, if we set this limit to 20, only 20 handlers will be occupied
and other 80 handlers can still handle requests to other RS. The availability of this service
is 90% as expected.

This message was sent by Atlassian JIRA

View raw message