hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Feng Honghua (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9467) write can be totally blocked temporarily by a write-heavy region
Date Fri, 13 Sep 2013 06:00:54 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766263#comment-13766263
] 

Feng Honghua commented on HBASE-9467:
-------------------------------------

Change and explanation of the patch:

1. Throw RegionOverloadedException immediately rather than wait/retry within HRegion when
the target region is above the memstore limit, this avoid write requests on region above memstore
limit occupying/saturating handler threads. This change is in HRegion.checkResources method.

2. Reuse the exception handling and retry mechanism of AsyncProcess in client to handle RegionOverloadedException
thrown from RS. Since RegionOverloadedException is not a DoNotRetryIOException, it'll be handled
the same way as other non-DoNotRetryIOException thrown from RS by AsyncProcess and the according
request will be retried using incremental backoff. 
   In a more general sense, we can view RegionOverloadedException as another kind of retriable
exception and reuse all the current handling for it in AsyncProcess/client, so no change in
client side code. And if we really want to use exponential backoff rather than incremental
backoff for RegionOverloadedException, as Todd suggested, we can change the code in AsyncProcess
accordingly.

3. We also need to check memstore limit and throw RegionOverloadedException for 'increment'
and 'append' operations, since they also insert kv to memstore and increase its size. (checkResources
is not called for these two operations in HRegion previously, corrected here)

4. In UT TestHFileArchiving, RegionOverloadedException is thrown during loadRegion and since
the 'put' operations are called directly via HRegion, not via client/AsyncProcess, a similiar
'catch-and-wait' handling is added here to proceed without failure.

[~nkeywal] / [~stack] / [~tlipcon] : Any feedback for the patch? Thanks in advance.
                
> write can be totally blocked temporarily by a write-heavy region
> ----------------------------------------------------------------
>
>                 Key: HBASE-9467
>                 URL: https://issues.apache.org/jira/browse/HBASE-9467
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Feng Honghua
>            Assignee: Feng Honghua
>         Attachments: HBASE-9467-trunk-v0.patch
>
>
> Write to a region can be blocked temporarily if the memstore of that region reaches the
threshold(hbase.hregion.memstore.block.multiplier * hbase.hregion.flush.size) until the memstore
of that region is flushed.
> For a write-heavy region, if its write requests saturates all the handler threads of
that RS when write blocking for that region occurs, requests of other regions/tables to that
RS also can't be served due to no available handler threads...until the pending writes of
that write-heavy region are served after the flush is done. Hence during this time period,
from the RS perspective it can't serve any request from any table/region just due to a single
write-heavy region.
> This sounds not very reasonable, right? Maybe write requests from a region can only be
served by a sub-set of the handler threads, and then write blocking of any single region can't
lead to the scenario mentioned above?
> Comment?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message