hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-17819) Reduce the heap overhead for BucketCache
Date Fri, 21 Jul 2017 12:46:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096195#comment-16096195
] 

Anoop Sam John edited comment on HBASE-17819 at 7/21/17 12:45 PM:
------------------------------------------------------------------

BlockCacheKey after align the heap overhead is 32 bytes and even after the change of ref to
byte it will be 32.  Still it will be worth doing that as above 32 GB heap size, compressed
ref wont be there and refs might take 8 bytes. Then it will make a diff.
BucketEntry was 64 bytes heap and after the patch it will be 48.
Also we will be removing 40 bytes per entry as we remove blocksByHFile Set.
So the math is 
Now -  32 + 64 + 40 + 40 = 176
After patch - 32 + 48 + 40 = 120

Tested with Java Instrumentation


was (Author: anoop.hbase):
BlockCacheKey after align the heap overhead is 32 bytes and even after the change of ref to
byte it will be 32.  Still it will be worth doing that as above 32 GB heap size, compressed
ref wont be there and refs might take 8 bytes. Then it will make a diff.
BucketEntry was 64 bytes heap and after the patch it will be 48.
Also we will be removing 40 bytes per entry as we remove blocksByHFile Set.
So the math is 
Now -  32 + 64 + 40 + 40 = 176
After patch - 32 + 48 + 40 = 120

> Reduce the heap overhead for BucketCache
> ----------------------------------------
>
>                 Key: HBASE-17819
>                 URL: https://issues.apache.org/jira/browse/HBASE-17819
>             Project: HBase
>          Issue Type: Sub-task
>          Components: BucketCache
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>
>         Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch
>
>
> We keep Bucket entry map in BucketCache.  Below is the math for heapSize for the key
, value into this map.
> BlockCacheKey
> ---------------
> String hfileName  -  Ref  - 4
> long offset  - 8
> BlockType blockType  - Ref  - 4
> boolean isPrimaryReplicaBlock  - 1
> Total  =  12 (Object) + 17 = 29
> BucketEntry
> ------------
> int offsetBase  -  4
> int length  - 4
> byte offset1  -  1
> byte deserialiserIndex  -  1
> long accessCounter  -  8
> BlockPriority priority  - Ref  - 4
> volatile boolean markedForEvict  -  1
> AtomicInteger refCount  -  16 + 4
> long cachedTime  -  8
> Total = 12 (Object) + 51 = 63
> ConcurrentHashMap Map.Entry  -  40
> blocksByHFile ConcurrentSkipListSet Entry  -  40
> Total = 29 + 63 + 80 = 172
> For 10 million blocks we will end up having 1.6GB of heap size.  
> This jira aims to reduce this as much as possible



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message