hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1460) Concurrent LRU Block Cache
Date Mon, 01 Jun 2009 06:18:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714989#action_12714989
] 

Jonathan Gray commented on HBASE-1460:
--------------------------------------

Thanks for review stack.

Posting new patch in a second.  To answer your questions:

- Class comments added
- setBuffer is currently used when an already cached block is cached again.  behavior is defined
in javadocs.  its assumed this will never happen though.
- align moved to Bytes class and unit test added!
- the get() method in the queue is used in the eviction algorithm, could actually use an iterator
instead would keep us from doing extra work in the queue
- yes, should just name it LRU just separate because i've not had a trunk with the recently
committed one yet
- comments removed (was keeping that around because i had to rework some of it into the new
constructor)
- new patch includes a growing unit test for the cache.  i'm planning on adding one or two
more methods but there are three tests which cover almost all functionality and it passes.
 I was seeing some failures on the third test, but then it went away with seemingly no change.
 I suspect there might be a race condition in the calling of the eviction thread.  The call
to the thread is heavily influenced by the solr implementation, dig in there if you could.

> Concurrent LRU Block Cache
> --------------------------
>
>                 Key: HBASE-1460
>                 URL: https://issues.apache.org/jira/browse/HBASE-1460
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>             Fix For: 0.20.0
>
>         Attachments: HBASE-1460-v1.patch
>
>
> The LRU-based block cache that will be committed in HBASE-1192 is thread-safe but contains
a big lock on the hash map.  Under high load, the block cache will be hit very heavily from
a number of threads, so it needs to be built to handle massive concurrency.
> This issue aims to implement a new block cache with LRU eviction, but backed by a ConcurrentHashMap
and a separate eviction thread.  Influence will be drawn from Solr's ConcurrentLRUCache, however
there are major differences because solr treats all cached elements as equal size whereas
we are dependent on our HeapSize interface with realistic (though approximate) heap usage.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message