hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elliott Clark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14978) Don't allow Multi to retain too many blocks
Date Tue, 15 Dec 2015 10:58:46 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057850#comment-15057850

Elliott Clark commented on HBASE-14978:

Yeah so we'll oom if we're bringing things on heap ( branch-1.2 ).
We'll kill the block cache by not allowing blocks to be free'd on other branches.

bq.The only thing that could happen now is that till all the Results could be transferred
to the client we may hold the block and so any such blocks cannot be evicted till the reference
is cleared.
Yeah. So if you have a high traffic server it's pretty easy to get to a state where this really
hurts. 30 (default min) Threads * 2 gb (easy to get to example) of data blocks makes for a
very unhappy block cache.

On heap vs off heap is not substantially changing the issue. We are holding onto datablocks
regardless of size needed or reserved. On heap we're encroaching into the memstore/blockcache
reserved space. Causing the JVM to run out of memory and have to either full GC or just OOME.

Off heap and we're encroaching into the index blocks and bloom blocks. Causing the block cache
to either run out of memory and be un-able to release anything, or to release the most important
index and bloom blocks.

In any scenario we're just moving around which subsystem of the regionserver is running out
of memory it thinks should be available.

> Don't allow Multi to retain too many blocks
> -------------------------------------------
>                 Key: HBASE-14978
>                 URL: https://issues.apache.org/jira/browse/HBASE-14978
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.0.0, 1.2.0, 1.3.0
>            Reporter: Elliott Clark
>            Assignee: Elliott Clark
>            Priority: Critical
>         Attachments: HBASE-14978-v1.patch, HBASE-14978-v2.patch, HBASE-14978.patch
> Scans and Multi's have limits on the total size of cells that can be returned. However
if those requests are not all pointing at the same blocks then the KeyValues can keep alive
a lot more data than their size.
> Take the following example:
> A multi with a list of 10000 gets to a fat row. Each column being returned in in a different
block. Each column is small 32 bytes or so.
> So the total cell size will be 32 * 10000 = ~320kb. However if each block is 128k then
total retained heap size will be almost 2gigs.

This message was sent by Atlassian JIRA

View raw message