hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
Date Wed, 27 Dec 2017 13:05:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304514#comment-16304514
] 

Eshcar Hillel commented on HBASE-18294:
---------------------------------------

My previous solution had three different counters in MemStoreSize: (1) one for data size (2)
one for on-heap size (3) one for off-heap size. 
The main reason I rolled back this change is the understanding that in the near future HBase
is going to support only clusters that are either entirely on-heap or entirely off-heap and
so there is no need to measure on- and off-heap occupancy separately in any level (cluster,
RS, region or memstore).
This means that when we say heap: heap size, heap occupancy, heap space, etc. we mean all
bytes allocated on the heap including JVM and OS heap.
If an implementation of the {{Cell}} class the method {{heapSize()}} does not return the number
of bytes allocated to store data that is related to this cell (key, value, metadata, etc.)
then I would say the implementation does not satisfy the specification of the method and needs
to be changed.

Is there any reason the method was implemented this way??

How about the following change adding {{KeyValueUtil.length(this)}} to the computation in
the implementation of {{ByteBufferKV}}
I think it will fix the problem. Do you see any reason it shouldn't work?
{code}
  public long heapSize() {
    if (this.buf.hasArray()) {
      return ClassSize.align(FIXED_OVERHEAD + length);
    }
    return ClassSize.align(FIXED_OVERHEAD) + KeyValueUtil.length(this);
  }
{code}

> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
>                 Key: HBASE-18294
>                 URL: https://issues.apache.org/jira/browse/HBASE-18294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>         Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch,
HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch,
HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, HBASE-18294.11.patch, HBASE-18294.11.patch,
HBASE-18294.12.patch, HBASE-18294.13.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the store to
another threshold (that can be configured with hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size (key-value only)
to the threshold where it should compare the heap size (which includes index size, and metadata).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message