hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1590) Extend TestHeapSize and ClassSize to do "deep" sizing of Objects
Date Tue, 30 Jun 2009 02:40:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725440#action_12725440
] 

Jonathan Gray commented on HBASE-1590:
--------------------------------------

I'm not sure we need to do this anymore.  Patch going in for HBASE-1591 cleaned up LruBlockCache
heapsizing and it works well now and is accurate.

Remaining issues are... 

- How do we really ensure sizing of the protected members of things like ConcurrentHashMap
(Entry and Segment).  Can use SizeOf but would rather try to do some hackery/reflection business
so we can dig in with ClassSize.
- Review of MemStore heapSize implementation... Same issue as above for ConcurrentSkipListMap

> Extend TestHeapSize and ClassSize to do "deep" sizing of Objects
> ----------------------------------------------------------------
>
>                 Key: HBASE-1590
>                 URL: https://issues.apache.org/jira/browse/HBASE-1590
>             Project: Hadoop HBase
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: Jonathan Gray
>             Fix For: 0.20.0
>
>
> As discussed in HBASE-1554 there is a bit of a disconnect between how ClassSize calculates
the heap size and how we need to calculate heap size in our implementations.
> For example, the LRU block cache can be sized via ClassSize, but it is a shallow sizing.
 There is a backing ConcurrentHashMap that is the largest memory consumer.  However, ClassSize
only counts that as a single reference.  But in our heapSize() reporting, we want to include
*everything* within that Object.
> This issue is to resolve that dissonance.  We may need to create an additional ClassSize.estimateDeep(),
we may need to rethink our HeapSize interface, or maybe just leave it as is.  The two primary
goals of all this testing is to 1) ensure that if something is changed and the sizing is not
updated, our tests fail, and 2) ensure our sizing is as accurate as possible.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message