hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1186) Memory-aware Maps with LRU eviction
Date Sat, 07 Feb 2009 22:59:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671535#action_12671535

Jonathan Gray commented on HBASE-1186:

I didn't have time to finish work on this yesterday.  I just posted up what I had finished
before I left, it's definitely not done yet.

Will properly format it, etc well before it's ready to be included.

Re: 32-bit vs 64-bit there are some tests that can be used.  My current plan is to go forward
with assuming 64bit and in the future we can easily add something to detect like you said.
 It would then just change the size of REFERENCE which would bubble up into anything implementing
HeapSize which should be using that static to compute its size.

There is some stuff left over from other code in there (transient datamembers and such) that
I will be taking out.  I see no need for serialization.  Same goes for making things private,
have been working on the lru and heapsize elements and not so much the rest of the code. 
Will clean it up next week.

Currently, maxMemUsage is passed in.  This LRU is intended for cell cache which I think should
be a per-regionserver setting.  This can be adapted to work for the block cache as well in
any number of ways.  There are a few reasons I don't think softrefs are the way to go for
block caching, definitely not the way to go for cell caching.  Next week I'll put together
the proposal for why I think we should use our own LRU mechanism rather than relying on softrefs
across the board.  The primary issues are how this will interact with existing memcache/heap
monitoring, unpredictability of softref eviction, and the ability to implement in-memory tables
very simply if we have our own lru mechanism (implemented in the way described in the bigtable

Same goes for synchronization.  This is an early version not ready for prime time.  Not sure
what my plan is yet will post thoughts here.  Will be working on this early in the week.

Thanks for the review stack.

> Memory-aware Maps with LRU eviction
> -----------------------------------
>                 Key: HBASE-1186
>                 URL: https://issues.apache.org/jira/browse/HBASE-1186
>             Project: Hadoop HBase
>          Issue Type: New Feature
>            Reporter: Jonathan Gray
>            Assignee: Erik Holstad
>            Priority: Critical
>             Fix For: 0.20.0
>         Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
> Caching is key for 0.20.  We need a set of memory-aware data structures to manage our
> I propose two initial classes:  LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache.  Erik Holstad
has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. 
It should try to use all available memory, but must contend with Memcaches so is resizable
to deal with heap pressure.  Adding high priority blocks (evicted last) gives us in-memory
functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message