hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15786) Create Offheap Memstore
Date Thu, 10 Nov 2016 05:42:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653107#comment-15653107

Anoop Sam John commented on HBASE-15786:

Oh ya.. I have added high level description but forgot to change the title of the jira. It
is not purely Off heap memstore.  The memstore as such is same.. It is a way to make Off heap
MSLAB pool.  You can see that is the major change.
The patch is big because I have made 2 renaming.
HeapMemorySizeUtil -> MemorySizeUtil.    Slight changes in methods here and some new methods.
The diff is this calc the off heap memory related stuff also. The max off heap memory for
global memstore.  
HeapMSLAB -> MSLABImpl.   This is because this deal with MSLAB on heap/off heap now.  Thought
of making a new class OffheapMSLAB.  The major change is when one say off heap MSLAB, we think
the chunks allocated by it is always off heap.  But we dont want create on demand DBBs which
will NOT get pooled.  Only those from the pool is DBB.  When pool is not having free chunk,
we will make new from MSLAB but that will be on heap in any case.  Is it good to call that
MSLAB as OffheapMALAB? I am thinking not.

A cell is getting copied into the chunk area while adding to memstore. We have write() API
in ExtendedCell.   So the cell is responsible for writing itself into the MSLAB chunk area.
 Only diff is this was taking byte[] as the chunk backing was that.  Now I changed chunk backing
to be BB.   So all the methods in CellUtil/KVUtil is copy of byte[] based methods.    That
is why confusing names. Did not do any renaming or refactor way.   Instead of cell doing the
write, we pull fragments of cell and write in KVUtil,  this is for the Cells NOT of type ExtendedCell..
 Actually speaking all cells in server side is supposed to be ExtendedCell type any way. 
Still for the completion of the impl, we consider other also.
bq.AN HRegion needs reference to RegionServerServices memory management?
This is existing ref. I did not add in this patch.

bq.Should the DefaultHeapMemoryTuner be renamed now you've renamed the class as MemorySizeUtil?
No.  This tuner tune only Heap memory.  When MSLAB pool is off heap we should not be tuning
that area.  That may be another tuner.  As of now we will keep it simple. Though we need some
changes. Ram is working on a patch for that. Will come soon.

> Create Offheap Memstore
> -----------------------
>                 Key: HBASE-15786
>                 URL: https://issues.apache.org/jira/browse/HBASE-15786
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>         Attachments: HBASE-15786.patch
> We can make use of MSLAB pool for this off heap memstore. 
> Right now one can specify the global memstore size (heap size) as a % of max memory using
a config. We will add another config with which one can specify the global off heap memstore
size. This will be exact size not as %. When off heap memstore in use, we will give this entire
area for the MSLAB pool and that will create off heap chunks. So when cells are added to memstore,
the cell data gets copied into the off heap MSLAB chunk spaces. Note that when the pool size
is not really enough and we need additional chunk creation, we wont use off heap area for
that.  We dony want to create so many on demand DBBs.

This message was sent by Atlassian JIRA

View raw message