hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Mon, 14 Mar 2016 09:28:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192967#comment-15192967
] 

Anastasia Braginsky commented on HBASE-14921:
---------------------------------------------

Hi Guys,

Long time no speak. Meanwhile, CellBlocksSegment based on simple Cells Array is ready (as
on second picture in the document). This means additional dereferencing and spending some
additional memory on object headers. I'm also implementing the totally flat CellBlocks, so
it would be possible to allocate it off-heap and it is generally more efficient (third picture
in the updated document). In order to do that I need a toBytes representation of a reference
to Chunk. As far as I know in pure Java you can’t get the address of an object. So it can
be done only by defining an index per Chunk and having a translation/mapping from some integer
index to Chunk and backward (from Chunk to its index).

Looking closer on HeapMemStoreLAB and MemStoreChunkPool I see that HeapMemStoreLAB.Chunk can
be allocated both directly from JVM Heap and also via MemStoreChunkPool, because MemStoreChunkPool
is sometimes not configured or can’t be used. It makes much more sense that all Chunk allocations
are centralized and go only through MemStoreChunkPool. It helps in managing the mapping described
above and also in off-heaping. Can MemStoreChunkPool exists always? MemStoreChunkPool can
pre-allocate Chunks as a Pool or allocate single Chunk on demand as HeapMemStoreLAB currently
does.

Then we have clear roles. HeapMemStoreLAB - deals only with reference counting and when Chunk
can be deallocated. It gets/returns chunks from/to MemStoreChunkPool. While MemStoreChunkPool
- deals only with chunks allocation, pre-allocation and re-allocation.
[~stack], [~anoop.hbase], and everybody, what do you think?

Thanks,
Anastasia

> Memory optimizations
> --------------------
>
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf
>
>
> Memory optimizations including compressed format representation and offheap allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message