hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
Date Tue, 28 Nov 2017 11:29:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268600#comment-16268600
] 

Anoop Sam John commented on HBASE-18294:
----------------------------------------

Ya as I said in above comments, we can continue to have decision for flush based on data size
OR heap Size.   FOr on heap cases (which is the one what we have till now) there will be no
differences at all.  Ya I agree to the concern that people will see diff wrt blocked writes
and so overall throughput.  
For off heap there is no need to do so many separate accounting and configs.  Because there
also any way we will have heap size based checks. Data size based check is what mainly govern
the flushes there. Because the whole work moves the heap overhead because of the data size
to off heap area. I dont know why this is not acceptable.
bq.There are claims pro reducing memstore size like for reducing GC, but there are also claims
pro increasing the size to reduce number of flushes, reduce number of compactions and reduce
write amplification.
Ya that is what. You can not say faster flush is better always. It all depends on the global
heap size available for the all regions.

My strong concern is against adding public APIs to set off heap flush size on heap flush size.
We were having only flush size been configured by user. As every thing in on heap, we ended
up deciding based on heap size.  That same can not be equated to exposing heap flush size
(flushSize ) and off heap flush size now.

> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
>                 Key: HBASE-18294
>                 URL: https://issues.apache.org/jira/browse/HBASE-18294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>         Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch,
HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the store to
another threshold (that can be configured with hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size (key-value only)
to the threshold where it should compare the heap size (which includes index size, and metadata).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message