hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
Date Fri, 03 Mar 2017 05:38:46 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893768#comment-15893768
] 

Anoop Sam John commented on HBASE-17338:
----------------------------------------

bq.Thanks. So, datasize vs heapsize, have we written these up? datasize is serialized size...
whether serialized on rpc or in blockcache and heapsize is how large a 'live' Cell in java
heap is? If the Cell data is offheap, it is the the onheap Cell proxy object?
Yes
bq.How does the global threshold work in the case where we are doing offheap accounting too.
The global check will look at the onheap limit and the offheap limit and if we hit the offheap
limit before we hit the onheap limit, we'll flush?
In case of off heap MSLAB in place, the global memstore size is specified using a new config
as u know.  So we will have this check against the aggregated dataSize .  Also we will be
having a max possible on heap memstore global size (Defualt 40% of xmx)  This check also done.
Any one of the condition met, we will do some forced flush. (What we do now also) with blocking
writes.
bq.Should be easy enough to do?
Should be IMO

> Treat Cell data size under global memstore heap size only when that Cell can not be copied
to MSLAB
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-17338
>                 URL: https://issues.apache.org/jira/browse/HBASE-17338
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>    Affects Versions: 2.0.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>
>         Attachments: HBASE-17338.patch, HBASE-17338_V2.patch, HBASE-17338_V2.patch, HBASE-17338_V4.patch,
HBASE-17338_V5.patch
>
>
> We have only data size and heap overhead being tracked globally.  Off heap memstore works
with off heap backed MSLAB pool.  But a cell, when added to memstore, not always getting copied
to MSLAB.  Append/Increment ops doing an upsert, dont use MSLAB.  Also based on the Cell size,
we sometimes avoid MSLAB copy.  But now we track these cell data size also under the global
memstore data size which indicated off heap size in case of off heap memstore.  For global
checks for flushes (against lower/upper watermark levels), we check this size against max
off heap memstore size.  We do check heap overhead against global heap memstore size (Defaults
to 40% of xmx)  But for such cells the data size also should be accounted under the heap overhead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message