hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15950) Fix memstore size estimates to be more tighter
Date Thu, 09 Jun 2016 23:29:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323594#comment-15323594

Enis Soztutar commented on HBASE-15950:

bq. So how the object size estimates looks like now? (For a KV added into CSLM)
In the test that I was doing, the JFR profiler shows 2.8G as used heap for memstore objects,
and we are estimating 2.9 (previously 4.5). Pretty close. 

bq. Use UnsafeAvailChecker#isAvailable() instead.

bq. These changes are also based on JOL
Yes. That particular change is because, we were not counting the object header size from OBJECT,
but by REFERENCE*2, and we were not aligning the array objects. 

bq. This is good to do, but I think the warning in the release note likely needs to be louder
and that it should also go in upgrade notes for 2.0.
Let me update the release notes with your suggestion. 

bq. Should we consider pairing this change with a reduction in the default memstore usage
I think it is fine to leave it as it is. 

> Fix memstore size estimates to be more tighter
> ----------------------------------------------
>                 Key: HBASE-15950
>                 URL: https://issues.apache.org/jira/browse/HBASE-15950
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 2.0.0
>         Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png, hbase-15950-v0.patch
> While testing something else, I was loading a region with a lot of data. Writing 30M
cells in 1M rows, with 1 byte values. 
> The memstore size turned out to be estimated as 4.5GB, while with the JFR profiling I
can see that we are using 2.8GB for all the objects in the memstore (KV + KV byte[] + CSLM.Node
+ CSLM.Index). 
> This obviously means that there is room in the write cache that we are not effectively

This message was sent by Atlassian JIRA

View raw message