hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2500) Block indexes and meta blocks should be put into the LRU
Date Wed, 28 Apr 2010 21:34:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861948#action_12861948

Jonathan Gray commented on HBASE-2500:

My preference would be to add all of these to the existing block cache LRU.  The open question
then would be whether they are treated like other normal data blocks, or whether we add additional
block types.  The design of the LRU actually makes it quite easy to add additional types (currently
there is single access block, multiple access block, and in-memory block), and there is not
much overhead involved.  The additional overhead is done in the background eviction thread
so doesn't add anything to the put/get methods.

If there was a new block type or two, like "index" and/or "meta", then these would be able
to have their own slice of the total block cache pie.  The LRU gives each of the different
block types their own percentage of the total LRU size.  Each is able to grow above it's slice
if others are under-utilized, the LRU just guarantees that during an eviction that block type
will be given at least the specified slice size.

I think we'll also need to add code to re-fetch block indexes / meta blocks if they are not
in the block cache.

> Block indexes and meta blocks should be put into the LRU
> --------------------------------------------------------
>                 Key: HBASE-2500
>                 URL: https://issues.apache.org/jira/browse/HBASE-2500
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Jonathan Gray
>             Fix For: 0.20.5, 0.21.0
> Currently we are not tracking the heap usage of block indexes.  We're just relying on
the fact that we leave a large percentage of total heap as part of "everything else" not in
memstores or the block cache.  The "everything else" portion should be left for ephemeral/transient
memory usage, not long-lived objects like block indexes.  Introduction of blooms will extend
this issue to meta blocks in general.
> All of this should be moved into an LRU.  Either the existing block cache LRU or a new

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message