Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1458942365.1234038419654.JavaMail.jira@brutus>
Date: Sat, 7 Feb 2009 12:26:59 -0800 (PST)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Commented: (HBASE-1186) Memory-aware Maps with LRU eviction
In-Reply-To: <1419416203.1233856679675.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671521#action_12671521 ] 

stack commented on HBASE-1186:
------------------------------

You think we should go to bother of trying to elicit if 32-bit vs. 64-bit?  (There is probably a canonical way but perhaps looking at bytes in an Integer would be a flag)?  When HeapSize is classloaded, it could run static methods to figure 32-bit vs. 64-bit and then adjust the defines in this class accordingly?  Then have methods on HeapSize that implementers call to size themselves for the different architectures?  Maybe thats something for later, for the embedded hbase (smile)?  Lets just go w/ 64-bit for now.

Let me know if you want me to run some tests on profiler to check your sizings.

For next version of LruHashMap, fix spacing -- its two spaces by default and no tabs  -- and add apache license.

Why do you have transient datamembers?  Are your Maps meant to be (java) serializable?

Make most of the data members private rather than default access?

So, maxMemUsage is passed in?  It'll be some percentage of the heap?

We're not soft referencing it because of your testing showing soft references are not LRU?

It'll be really interesting watching this thing at core of a concurrent app.  There are no synchronizes anywhere.  Whats the plan for that?

Good stuff


> Memory-aware Maps with LRU eviction
> -----------------------------------
>
>                 Key: HBASE-1186
>                 URL: https://issues.apache.org/jira/browse/HBASE-1186
>             Project: Hadoop HBase
>          Issue Type: New Feature
>            Reporter: Jonathan Gray
>            Assignee: Erik Holstad
>            Priority: Critical
>             Fix For: 0.20.0
>
>         Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20.  We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes:  LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache.  Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61.  It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure.  Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.