hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore
Date Sun, 18 Dec 2016 13:39:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758850#comment-15758850

Anastasia Braginsky commented on HBASE-16421:

Hey Again,

Here is the road-map for how we see the CellChunkMap is going to be added to HBase.
We assume two prerequisites:
1. We need evaluation of how reasonable it would be to use CellChunkMap with the scans due
to the GC. It is going to be accomplished soon by [~ram_krish] and we will continue by now
assuming that CellChunkMaps are valid anyway.
2. The Cell will have the API to get its chunk ID. This will be resolved by [~anoop.hbase]
or [~ram_krish] under HBASE-16437 and HBASE-16438. And you will just let us know what the
API is. What is the status there by the way?

In the following each task is going to be a JIRA, so we try to split the work and have smaller
patches to manage, review, and commit.
*1. MemStoreChunkAllocator Abstraction* Refactoring the separation between MemStoreChunkPool
and (new) MemStoreChunkAllocator. The latter allocates chunks either from heap or from pool
and assigns them the Chunk IDs. MemStoreChunkAllocator stores the mapping between chunks and
their IDs.
*2. CellChunkMap Implementation* Unit-tested only. Not user-observable. Bringing the code
from early versions of HBASE-14921 and adopting to the new and updated environment.
*3. Flattening segment index (with regular cells) into CellChunkMap upon in-memory flush*
Upon in-memory-flush if the memstore is compacting (CompactingMemStore) and off-heap MSLAB
storage then use CellChunkMap as a flattened index. Constraints: flattening only for regular
cells (no upserted cells, no big cells). Just assume all cells are regular, the support for
irregular cells will come in following JIRAs.
*4. Copying the upserted cells upon in-memory flush* In time of the in-memory flush, when
a normal-size cell is encountered which is not allocated on MSLAB (due to upsert) copy it
to the MSLAB normal-size chunk. 
*5. Support for variable-size chunks in MemStoreChunkAllocator* For future big cells support.
Those chunks are allocated on-heap upon request and released when unneeded - no pool usage.
Assuming there are not much such threads. If the big cells are a common case for some workload,
consider just increasing the chunk's size to stay off-heap.
*6. Support for flattening with irregular cells*

Any comments?

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> ------------------------------------------------------------------------------
>                 Key: HBASE-16421
>                 URL: https://issues.apache.org/jira/browse/HBASE-16421
>             Project: HBase
>          Issue Type: Umbrella
>            Reporter: Anastasia Braginsky
>         Attachments: CellChunkMapRevived.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include all the parts
of integration of the CellChunkMap to the MemStore.

This message was sent by Atlassian JIRA

View raw message