hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
Date Sun, 26 Nov 2017 08:47:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265945#comment-16265945

Eshcar Hillel commented on HBASE-18294:

Ram and Anoop, the reason we see so much heap global pressure is that the regions themselves
are not conservative enough to make flush decisions early on. *Changing default values is
not a way to fix this inherent problem*:
(1) Reducing the threshold may solve the problem for some setting but will not solve it for
other settings. For example, in the same experiment if we have the threshold set to 64MB but
with twice as much regions we will see the same affect.
(2) There are claims pro reducing memstore size like for reducing GC, but there are also claims
pro increasing the size to reduce number of flushes, reduce number of compactions and reduce
write amplification.
(3) In addition, even if we change the default values the system should have optimal performance
with the values set by the admin which can be any number.

The core changes in this patch focus on the mechanism and decision making for region level
flushes, namely evaluate total heap size instead of data size only. 
The changes at the region server accounting level are mainly cosmetic changes, to make on-heap
and off-heap symmetric (why should we ignore the CCM index when it is allocated off-heap,
even if it is small, if we can count it the same way we count the CAM index for on-heap?).
And I think the changes are not that dramatic about 20 lines of code in {{RegionServerAccounting}},
they do not complicate things much.

Can we in-parallel to the discussion here continue with concrete comments on the code in RB
so we can converge towards commit. 

> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>                 Key: HBASE-18294
>                 URL: https://issues.apache.org/jira/browse/HBASE-18294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>         Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch,
HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch
> A region is flushed if its memory component exceed a threshold (default size is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the store to
another threshold (that can be configured with hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size (key-value only)
to the threshold where it should compare the heap size (which includes index size, and metadata).

This message was sent by Atlassian JIRA

View raw message