hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5162) Basic client pushback mechanism
Date Sat, 18 Feb 2012 05:00:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210809#comment-13210809

jiraposter@reviews.apache.org commented on HBASE-5162:

This is an automatically generated e-mail. To reply, visit:


    I find this a bit dubious.
    This won't actually slow the client thread down, but just accumulate more data and reduce
the number of RPCs. In the end it might lead to more load on the server, because we can deliver
more puts as with fewer but larger batches.
    I'd rather just rely on the server sleeping the thread for a bit (as you do later).


    What if the flusher is not null? Should we re-calculate the wait time?


    If sleepTime is 0 (for example from NoServerBackoffPolicy), we should probably not create
the thread and flush right here.
    (But as I said in the comment above, I'd probably not bother with this extra thread to
begin with :) )


    Was this a problem before? Or only now becaue of the background thread?


    This will break backwards compatibility, right? (Not saying that's not ok, just calling
it out)
    I'd almost rather have the client not know about this, until we reach a bad spot (in which
case we can throw back retryable exceptions).


    Ah ok, this is where we gracefully delay the server thread a bit.
    Seems this would need to be tweaked carefully to make it effective while not slowing normal
    Should the serverPauseTime be somehow related to the amount of pressure.
    I.e. wait a bit more if the pressure is higher?
    Maybe the pausetime calculation should be part of the pluggable policy?
    Also in the jira there was some discussion about throwing (presumably retryable) exceptions
back to the client is the pressure gets too high. That would slow the client, without consuming
server resources (beyond multiple requests).


    General comment: Where are we on putting/documenting these things in hbase-defaults.xml?

- Lars

On 2012-02-16 20:45:50, Jesse Yates wrote:
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3930/
bq.  -----------------------------------------------------------
bq.  (Updated 2012-02-16 20:45:50)
bq.  Review request for hbase, Michael Stack, Jean-Daniel Cryans, and Lars Hofhansl.
bq.  Summary
bq.  -------
bq.  Under heavy write load, HBase will create a saw-tooth pattern in accepting writes. This
is due to the I/O in minor compactions not being able to keep up with the write load. Specifically,
the memstore is attempting to flush while we are attempting to do a minor compaction, leading
to blocking _all_ writes. Instead, we need to have the option of graceful degradation mechanism.
bq.  This patch supports both a short-term,adjustable server-side write blocking as well as
client-side back-off to help alleviate temporary memstore pressure.
bq.  This addresses bug HBASE-5162.
bq.      https://issues.apache.org/jira/browse/HBASE-5162
bq.  Diffs
bq.  -----
bq.    src/main/java/org/apache/hadoop/hbase/HConstants.java 763fe89 
bq.    src/main/java/org/apache/hadoop/hbase/client/BackoffPolicy.java PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/client/HTable.java 57605e6 
bq.    src/main/java/org/apache/hadoop/hbase/client/MonitoredResult.java PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/client/NoServerBackoffPolicy.java PRE-CREATION

bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 25cb31d 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7d7be3c 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/MemstorePressureMonitor.java PRE-CREATION

bq.    src/main/java/org/apache/hadoop/hbase/regionserver/OperationStatus.java 1b94ab5 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/PressureMonitor.java PRE-CREATION

bq.    src/main/java/org/apache/hadoop/hbase/regionserver/SimpleMemStorePressureMonitor.java
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/StoreUtils.java PRE-CREATION 
bq.    src/test/java/org/apache/hadoop/hbase/client/TestClientBackOff.java PRE-CREATION 
bq.    src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStorePressureMonitor.java
bq.  Diff: https://reviews.apache.org/r/3930/diff
bq.  Testing
bq.  -------
bq.  Thanks,
bq.  Jesse

> Basic client pushback mechanism
> -------------------------------
>                 Key: HBASE-5162
>                 URL: https://issues.apache.org/jira/browse/HBASE-5162
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.92.0
>            Reporter: Jean-Daniel Cryans
>             Fix For: 0.94.0
>         Attachments: java_HBASE-5162.patch
> The current blocking we do when we are close to some limits (memstores over the multiplier
factor, too many store files, global memstore memory) is bad, too coarse and confusing. After
hitting HBASE-5161, it really becomes obvious that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
>  - Send some exception to the client, like OverloadedException, that's thrown when some
situation happens like getting past the low memory barrier. It would be thrown when the client
gets a handler and does some check while putting or deleting. The client would treat this
a retryable exception but ideally wouldn't check .META. for a new location. It could be fancy
and have multiple levels of pushback, like send the exception to 25% of the clients, and then
go up if the situation persists. Should be "easy" to implement but we'll be using a lot more
IO to send the payload over and over again (but at least it wouldn't sit in the RS's memory).
>  - Send a message alongside a successful put or delete to tell the client to slow down
a little, this way we don't have to do back and forth with the payload between the client
and the server. It's a cleaner (I think) but more involved solution.
> In every case the RS should do very obvious things to notify the operators of this situation,
through logs, web UI, metrics, etc.
> Other ideas?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message