hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-1460) Concurrent LRU Block Cache
Date Wed, 24 Jun 2009 21:41:07 GMT

     [ https://issues.apache.org/jira/browse/HBASE-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jonathan Gray updated HBASE-1460:

    Attachment: HBASE-1460-v3.patch

Addresses all issues from last comment, stack.

Renamed things to be proper.  Replaces LruBlockCache instead of adding a new class.  Added
more javadocs, integrated with metrics, etc.

Regarding the timing issue in the tests, it will take a bit of a rework so that the eviction
thread "reports" back when it's done or some such thing.  Rather complex.  I've changed things
around a bit so that I can disable the thread and run them from the context of the test rather
than in the background thread.  So the tests that are testing the evictions themselves, heap
size, etc.. will not be tied to any timing issues.  However, the issue there is we aren't
testing the operation of the background thread anymore.  So I added a single test that just
checks if the background thread works (it contains the same kind of timing issue).  However,
in this case I really don't think we'll have a problem on hudson, the eviction is very fast
and is executed directly.

I also added a scheduled thread inside that outputs statistics on a fixed period.  It's really
just in there for now while I do some heavy debugging, can be taken out before commit (or
left in if people think it's useful, probably a better way to do it though).

This breaks erik's new TestHeapSize.  Need to talk to him tomorrow about how we get that working.
 The cache heap size that it reports includes ALL overhead not actually part of the blocks
(the pre-allocated Map, it's entries, etc).  TestLruBlockCache does a bunch of validation
of heapSize() reporting, but is more to ensure it's tracking block sizes rather than verifying
our static overheads.

Not ready for commit.  I want to add a few more tests and do some load testing.  Will report
more tomorrow.

> Concurrent LRU Block Cache
> --------------------------
>                 Key: HBASE-1460
>                 URL: https://issues.apache.org/jira/browse/HBASE-1460
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>             Fix For: 0.20.0
>         Attachments: HBASE-1460-v1.patch, HBASE-1460-v2.patch, HBASE-1460-v3.patch
> The LRU-based block cache that will be committed in HBASE-1192 is thread-safe but contains
a big lock on the hash map.  Under high load, the block cache will be hit very heavily from
a number of threads, so it needs to be built to handle massive concurrency.
> This issue aims to implement a new block cache with LRU eviction, but backed by a ConcurrentHashMap
and a separate eviction thread.  Influence will be drawn from Solr's ConcurrentLRUCache, however
there are major differences because solr treats all cached elements as equal size whereas
we are dependent on our HeapSize interface with realistic (though approximate) heap usage.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message