hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
Date Mon, 20 Nov 2017 13:07:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259214#comment-16259214

Eshcar Hillel commented on HBASE-18294:

[~anoopsamjohn] I hope you don't mind I'm adding your comment on the new off-heap flush size
configuration property here since it would be easier to have this important discussion in
the Jira rather in RB:
bq. Do we really need to add this extra flush size? It will be very difficult for the users
to set all. They used to set the flush size. Being a normal user, I set some flush size means
am expecting the flush will happen after the data reached this size.  We have implemented
this as heapSize oriented in the past as every thing was in heap. Also we can not make the
heap size of all memstores to grow beyond a limt. We have the global heap upper barrier and
stuff like that.
bq. The change for 2.0 is to change the per region flush decision to be data size based not
heap size based. I agree that is a change in the way we were working. So the flushes will
be delayed than the prev tuned ways.
bq. Also I agree to one of ur args (In mail chain I guess) that when we select regions to
flush because of global heap pressure, we should select the one with max heap size. Right
now that is not happening.
bq. So this directs to the need to track the data size and heap size both at region level.
bq. We can still have the flush decision at region level based on the data size only?  When
the global heap size is at barrier, we will select region based on the heap size. When the
data size barrier breaches(In off heap case), we will select regions based data size. 
bq. I dont think we have to do all these of adding one more off heap flush size etc. Pls lets
keep simple.

> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>                 Key: HBASE-18294
>                 URL: https://issues.apache.org/jira/browse/HBASE-18294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>         Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch,
HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch
> A region is flushed if its memory component exceed a threshold (default size is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the store to
another threshold (that can be configured with hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size (key-value only)
to the threshold where it should compare the heap size (which includes index size, and metadata).

This message was sent by Atlassian JIRA

View raw message