hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Mon, 28 Mar 2016 14:55:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214273#comment-15214273

Anastasia Braginsky commented on HBASE-14921:

Thanks [~ram_krish], you raise the really good questions about ChunkPool!

bq. So the CellBlockSerialized (I think we can have a better name) - will work only if the
MSLAB Chunk pool is ON right? If there is no MSLAB chunk pool still it can work? I may be
missing something.

Indeed this requires ChunkPool to be ON. Please pay attention that we can design ChunkPool
not to pre-allocate Chunks if this is what configuration wants. It can be just a global bookkeeper
for all allocated Chunks. We just require ChunkPool not to be NULL.

bq. Now in your shallowCellsToBuffer even the shallowBuffer is created with the same 'chunkSize'
as we create the deepbuffer also? Is that really needed? 

The “chunkSize” used in shallowCellsToBuffer holds the size of any Chunk created by ChunkPool.
I assume (as it is now in MemStoreChunkPool) that all Chunks created from the same instance
of Pool are of same size. This is why one can reuse the Chunks. If Chunks are "tailor-made"
with different sizes, they almost cannot be reused.  

bq. How does things work when the chunkpool gets reset and initialized? 

This would be a problem. What is the case when ChunkPool need to be reset and initialized?


And BTW regarding the names, do you like the names that I have suggested above?

Regarding the names, as “CellBlock” is already in some other use, we suggest the following
names (almost as you suggested).

CellBlock -> CellFlatMap

CellBlockObjectArray -> CellArrayMap

CellBlockSerialized -> CellChunkMap

Why “flat”? Because this is how we call the process of changing the ConcurrentSkipListMap
to CellMap. When we have in-memory-flash and there is no need to compact (unnecessary copies)
we flatten the segment. Meaning we create ImmutableSegment with the same MSLAB, but with different
Map, replacing ConcurrentSkipListMap to CellMap. What do you think?


> Memory optimizations
> --------------------
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch
> Memory optimizations including compressed format representation and offheap allocations

This message was sent by Atlassian JIRA

View raw message