hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
Date Thu, 02 Mar 2017 22:24:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893152#comment-15893152

stack commented on HBASE-17338:

bq. Not sure whether used dataSize/cellSize here and there. Its data size only which is the
way as u said above. There is no cellSize as such. There is only cell heap size which is the
heap size (total) occupied by the cell impl object

Thanks. So, datasize vs heapsize, have we written these up? datasize is serialized size...
whether serialized on rpc or in blockcache and heapsize is how large a 'live' Cell in java
heap is? If the Cell data is offheap, it is the the onheap Cell proxy object?

bq. Ya there is one global threshold in both cases. Moreover in case of off heap, there is
a heap global threshold also. ie. By def 40% of xmx above which we will force flush. In case
of offheap, we have to check this extra thing also or else there is possibility of global
memstore size getting oversized and GC impacts/OOME.

How does the global threshold work in the case where we are doing offheap accounting too.
 The global check will look at the onheap limit and the offheap limit and if we hit the offheap
limit before we hit the onheap limit, we'll flush?

bq. Say in RS MSLAB is ON. But the regions from this particular table wont use MSLAB at all.

Should be easy enough to do?

Thanks [~anoop.hbase]

> Treat Cell data size under global memstore heap size only when that Cell can not be copied
> ---------------------------------------------------------------------------------------------------
>                 Key: HBASE-17338
>                 URL: https://issues.apache.org/jira/browse/HBASE-17338
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>    Affects Versions: 2.0.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>         Attachments: HBASE-17338.patch, HBASE-17338_V2.patch, HBASE-17338_V2.patch, HBASE-17338_V4.patch,
> We have only data size and heap overhead being tracked globally.  Off heap memstore works
with off heap backed MSLAB pool.  But a cell, when added to memstore, not always getting copied
to MSLAB.  Append/Increment ops doing an upsert, dont use MSLAB.  Also based on the Cell size,
we sometimes avoid MSLAB copy.  But now we track these cell data size also under the global
memstore data size which indicated off heap size in case of off heap memstore.  For global
checks for flushes (against lower/upper watermark levels), we check this size against max
off heap memstore size.  We do check heap overhead against global heap memstore size (Defaults
to 40% of xmx)  But for such cells the data size also should be accounted under the heap overhead.

This message was sent by Atlassian JIRA

View raw message