Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 24 Feb 2017 09:25:44 +0000 (UTC)
From: "Anoop Sam John (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13029131.1482153529000.54062.1487928344113@Atlassian.JIRA>
In-Reply-To: <JIRA.13029131.1482153529000@Atlassian.JIRA>
References: <JIRA.13029131.1482153529000@Atlassian.JIRA> <JIRA.13029131.1482153529589@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17338) Treat Cell data size under global
 memstore heap size only when that Cell can not be copied to MSLAB
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 24 Feb 2017 09:44:47 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882303#comment-15882303 ] 

Anoop Sam John commented on HBASE-17338:
----------------------------------------

Thanks for the reviews Stack & Ram.. I have to fix the failed test as those were having assert on heapOverhead size after cell addition to Memstore. Now we track heapSize not overhead.    Will add more comments as Stack asked
Regarding Ram's comment.
I have different opinion. I dont think we need change this way.   We track dataSize (irrespective of cell data in on heap or off heap area).. This dataSize been used at Segment level for in memory flush decision, Region level for on disk flushes and globally to force flush some regions.  At the 1st 2 levels, it is not doubt that we have to track all the cell data size together.  Now the point Ram says is when we have off heap configured and max off heap global size is say 12 GB,  once the data size globally reaches this level, we will force flush some regions.  So his point is for this tracking, we have to consider only off heap Cells and on heap Cell's data size should not get accounted in the data size but only in the heapSize. (At global level.  But at region and segment level it has to get applied).  2 reasons why I am not in favor of this
1. This makes the impl so complex.  We need to add isOffheap check down the layers.  Also at 2 layers we have to consider these on heap cell data size and one level not.
2. When off heap is enabled, (We have the MSLAB pool off heap in place), we will end up in having on heap Cells when the pool is out of BBs.  We will create on demand LABs on heap.  If we dont consider those cell's data size at global level, we may reach forced flush level a bit late.  That is the only gain.  But here the on demand LAB creation is a sign that the write load is so high and delaying the flush will add more pressure to the MSLAB and more and more on demand BBs (2 MB sized) need to get created.  One aim of the off heap work is to reduce the max heap space need for the servers.   

So lets consider the cell data size globally also (how we do now) and make global flushes.
Now even if MSLAB is used, the append/increment use cases wont use MSLAB. The cells will be on heap then.  But for such a use case, enabling MSLAB (and pool) is totally waste. That is a mis configuration.

More and more on demand BB creation, when MSLAB pool is a bad sign.  We have a WARN log for just one time as of now.. May be we should repeat this log at certain intervals. (Like for every 100th pool miss or so.)
We should be able to turn MSLAB usage ON/OFF per table also.  Now this is possible? Am not sure.  
These 2 things need to be checked and done in another jiras IMO.

> Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-17338
>                 URL: https://issues.apache.org/jira/browse/HBASE-17338
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>    Affects Versions: 2.0.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>
>         Attachments: HBASE-17338.patch, HBASE-17338_V2.patch, HBASE-17338_V2.patch, HBASE-17338_V4.patch
>
>
> We have only data size and heap overhead being tracked globally.  Off heap memstore works with off heap backed MSLAB pool.  But a cell, when added to memstore, not always getting copied to MSLAB.  Append/Increment ops doing an upsert, dont use MSLAB.  Also based on the Cell size, we sometimes avoid MSLAB copy.  But now we track these cell data size also under the global memstore data size which indicated off heap size in case of off heap memstore.  For global checks for flushes (against lower/upper watermark levels), we check this size against max off heap memstore size.  We do check heap overhead against global heap memstore size (Defaults to 40% of xmx)  But for such cells the data size also should be accounted under the heap overhead.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)