hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicolas Liochon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9467) write can be totally blocked temporarily by a write-heavy region
Date Tue, 10 Sep 2013 23:13:52 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13763677#comment-13763677
] 

Nicolas Liochon commented on HBASE-9467:
----------------------------------------

bq. Isn't the story worse than you paint?

Yeah. But for the WAL, I don't care that much: whatever we do, it's at the minimum network
call. Even if I add memory, SSD drives, whatever, it will be expensive. gets into the cache
(ours or OS caches) that do not require any i/o are a different animal.


bq.  What was the concurrency like in your server? tens or hundreds of handlers/readers?
500 threads doing gets like crazy
10 readers, the handlers were not called as I was doing the call from the reader.

Basically, if the load is CPU bound (a search in a cache), having more threads than core is
nearly useless (nearly because we have some i/o in the sockets).


The main issue around this test is that the performances depends a lot on the cpu cache (hence
the impr)ovement I got when I used the disruptor). But it's difficult to estimate how much
you're crashing the cache on a real workload.

Here I wanted to propose a kind of "fast path" for the operations that are likely to take
no i/o. But the result of the attempt was not great. I will try again (merging this with the
disruptor). I wanted to merge the responder call as well but I failed.

I'm also trying a (very simple) rowkey cache, and this seems to help more. Stay tuned for
this one ;-)

                
> write can be totally blocked temporarily by a write-heavy region
> ----------------------------------------------------------------
>
>                 Key: HBASE-9467
>                 URL: https://issues.apache.org/jira/browse/HBASE-9467
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Feng Honghua
>            Priority: Minor
>
> Write to a region can be blocked temporarily if the memstore of that region reaches the
threshold(hbase.hregion.memstore.block.multiplier * hbase.hregion.flush.size) until the memstore
of that region is flushed.
> For a write-heavy region, if its write requests saturates all the handler threads of
that RS when write blocking for that region occurs, requests of other regions/tables to that
RS also can't be served due to no available handler threads...until the pending writes of
that write-heavy region are served after the flush is done. Hence during this time period,
from the RS perspective it can't serve any request from any table/region just due to a single
write-heavy region.
> This sounds not very reasonable, right? Maybe write requests from a region can only be
served by a sub-set of the handler threads, and then write blocking of any single region can't
lead to the scenario mentioned above?
> Comment?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message