hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18010) Connect CellChunkMap to be used for flattening in CompactingMemStore
Date Thu, 18 May 2017 11:32:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015614#comment-16015614

Anastasia Braginsky commented on HBASE-18010:

While writing the integration I've encountered the following issue with the sizes. Generally,
the issue is that KeyValue.FIXED_OVERHEAD, meaning the size of the Cell object is not counted
as part of the any size. I am looking how size of the cells in segment is calculated:

protected void updateMetaInfo(Cell cellToAdd, boolean succ, boolean mslabUsed,
      MemstoreSize memstoreSize) {
    long cellSize = 0;
    // If there's already a same cell in the CellSet and we are using MSLAB, we must count
in the
    // MSLAB allocation size as well, or else there will be memory leak (occupied heap size
    // than the counted number)
    if (succ || mslabUsed) {
      cellSize = getCellLength(cellToAdd);
    long heapSize = heapSizeChange(cellToAdd, succ);
    incSize(cellSize, heapSize);
    if (memstoreSize != null) {
      memstoreSize.incMemstoreSize(cellSize, heapSize);
    minSequenceId = Math.min(minSequenceId, cellToAdd.getSequenceId());
    // In no tags case this NoTagsKeyValue.getTagsLength() is a cheap call.
    // When we use ACL CP or Visibility CP which deals with Tags during
    // mutation, the TagRewriteCell.getTagsLength() is a cheaper call. We do not
    // parse the byte[] to identify the tags length.
    if (cellToAdd.getTagsLength() > 0) {
      tagsPresent = true;

However, getCellLength() is implemented just as KeyValueUtil.length(cell) which doesn't include
counting Cell object size. I remember I was told that KeyValue.heapSize() is responsible for
counting both Cell "true" length and Cell object size. This is true, but KeyValue.heapSize()
is not on the path of updating the segment size. From here I see that the size of CellArrayMap
overhead is extremely small...

// REFERENCE in the CellArrayMap all the rest is counted in KeyValue.heapSize()

[~anoop.hbase], [~ramkrishna], what do you think about it? Am I missing something?

> Connect CellChunkMap to be used for flattening in CompactingMemStore
> --------------------------------------------------------------------
>                 Key: HBASE-18010
>                 URL: https://issues.apache.org/jira/browse/HBASE-18010
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Anastasia Braginsky
> The CellChunkMap helps to create a new type of ImmutableSegment, where the index (CellSet's
delegatee) is going to be CellChunkMap. No big cells or upserted cells are going to be supported

This message was sent by Atlassian JIRA

View raw message