hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
Date Thu, 28 Dec 2017 10:51:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305325#comment-16305325

Eshcar Hillel commented on HBASE-18294:

Thanks [~stack] for the document, I think its a good summary of the issue.

I think it is important to have symmetric behavior for on-heap and off-heap clusters. Even
if users do special tuning for off-heap clusters it is not reasonable to expect them understand
that on-heap flushes when total occupancy exceeds 128MB and off-heap flushes when key-values
are exceeding 128MB. 
I would like to avoid checking heapsize >128MB OR datasize > 128MB since this is exactly
defining unsymmetrical behavior for on-heap and off-heap clusters.

I understand also that [~anoop.hbase] and [~ram_krish] are objecting to change the implementation
of {{heapSize()}} so it accounts also for off-heap size, either because they feel it is confusing
or because they think it will cause some real problems in the off-heap implementation

So here is a suggestion: 
(1) I will change the name of the counter in {{MemStoreSize}} from {{heapSize}} to something
like {{totalAllocationSize}} or {{totalOccupancy}}  
(2) I will add methods {{estimatedTotalOccupancyOf(final Cell cell)}} and {{totalOccupancy()}}
to compute total occupancy of data+metadata no matter were they are allocated java-heap or
off-heap (the way I suggested above) and will use them to update the counters in {{MemStoreSize}}
(3) only changing region level flush decision, not changing RS global accounting and decisions

If this is acceptable, I can make a new patch in a day or two.

> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>                 Key: HBASE-18294
>                 URL: https://issues.apache.org/jira/browse/HBASE-18294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>             Fix For: 2.0.0-beta-2
>         Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch,
HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch,
HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, HBASE-18294.11.patch, HBASE-18294.11.patch,
HBASE-18294.12.patch, HBASE-18294.13.patch
> A region is flushed if its memory component exceed a threshold (default size is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the store to
another threshold (that can be configured with hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size (key-value only)
to the threshold where it should compare the heap size (which includes index size, and metadata).

This message was sent by Atlassian JIRA

View raw message