hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-15950) Fix memstore size estimates to be more tighter
Date Wed, 08 Jun 2016 02:41:21 GMT

     [ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Enis Soztutar updated HBASE-15950:
    Attachment: hbase-15950-v0.patch

Here is an initial patch without JOL dependency. We use Unsafe to estimate the header sizes
and oop reference size similar to the way JOL does it. For the changed ones, the estimates
matches what JOL reports. 

Apart from small cell size tests where it gives >50% improvement in available memstore
space, I've put this patch up in a YCSB cluster (single node). Inserted 30M rows of 1K size,
with and without the patch with 50 regions. The number of flushes decreased from 610 to 500,
and compactions from 64 to 41. Overall %20 decrease in flush / compaction IO is pretty significant.

> Fix memstore size estimates to be more tighter
> ----------------------------------------------
>                 Key: HBASE-15950
>                 URL: https://issues.apache.org/jira/browse/HBASE-15950
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 2.0.0
>         Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png, hbase-15950-v0.patch
> While testing something else, I was loading a region with a lot of data. Writing 30M
cells in 1M rows, with 1 byte values. 
> The memstore size turned out to be estimated as 4.5GB, while with the JFR profiling I
can see that we are using 2.8GB for all the objects in the memstore (KV + KV byte[] + CSLM.Node
+ CSLM.Index). 
> This obviously means that there is room in the write cache that we are not effectively

This message was sent by Atlassian JIRA

View raw message