hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Latham (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15950) Fix memstore size estimates to be more tighter
Date Wed, 08 Jun 2016 13:38:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320566#comment-15320566
] 

Dave Latham commented on HBASE-15950:
-------------------------------------

This is good to do, but I think the warning in the release note likely needs to be louder
and that it should also go in upgrade notes for 2.0.

Something like:
The estimates of heap usage by the memstore have been made more accurate, resulting in them
dropping by 10-50% in practice.  As a result, the actual heap usage of the memstore before
being flushed may increase by up to 100%.  If configured memory limits for the region server
had been tuned based on observed usage, this change could result in worse GC behavior or even
OutOfMemory errors.

Should we consider pairing this change with a reduction in the default memstore usage percent?



> Fix memstore size estimates to be more tighter
> ----------------------------------------------
>
>                 Key: HBASE-15950
>                 URL: https://issues.apache.org/jira/browse/HBASE-15950
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 2.0.0
>
>         Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png, hbase-15950-v0.patch
>
>
> While testing something else, I was loading a region with a lot of data. Writing 30M
cells in 1M rows, with 1 byte values. 
> The memstore size turned out to be estimated as 4.5GB, while with the JFR profiling I
can see that we are using 2.8GB for all the objects in the memstore (KV + KV byte[] + CSLM.Node
+ CSLM.Index). 
> This obviously means that there is room in the write cache that we are not effectively
using. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message