hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Wed, 20 Apr 2016 04:49:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249241#comment-15249241
] 

Anoop Sam John commented on HBASE-14921:
----------------------------------------

For the seqId I mean this
{code}
public Cell maybeCloneWithAllocator(Cell cell) {
    if (getMemStoreLAB() == null) {
      return cell;
    }

    int len = KeyValueUtil.length(cell);
    ByteRange alloc = getMemStoreLAB().allocateBytes(len);
    if (alloc == null) {
      // The allocation was too large, allocator decided
      // not to do anything with it.
      return cell;
    }
    assert alloc.getBytes() != null;
    KeyValueUtil.appendToByteArray(cell, alloc.getBytes(), alloc.getOffset());
    KeyValue newKv = new KeyValue(alloc.getBytes(), alloc.getOffset(), len);
    newKv.setSequenceId(cell.getSequenceId());
    return newKv;
  }
{code}
So the seqId is a long state on the Cell object. (We make KeyValue only here)
What I was saying is we will have a new impl for Cell which dont have seqId state. We will
ask Allocator to give cell length + 8 bytes and keep the seqId as the last 8 bytes after the
cell key and value part.  So when off heap MSLAB in place, we can keep these 8 bytes also
off heap.   And then this can be a special cell which can return the ref to MSLAB chunk so
that when we flatten this CSLM as part of in memory flush/compaction, we dont have to keep
the Cell object as such and we can get ref to MSLAB chunk and offset there.   Or may be we
can even have this Cell to have the chunkId too..  That will make us same way as 8 bytes per
cell meta data. (chunkId + offset)

Any way we can do those as a follow up issue.

> Memory optimizations
> --------------------
>
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch, HBASE-14921-V03.patch, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message