accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-3982) Add LRU mechanism to address memory concerns and possibly pre-fetch Tablet Locations.
Date Tue, 01 Sep 2015 00:04:46 GMT


Josh Elser commented on ACCUMULO-3982:

bq. It took about 0.5G. I also did experiments with a ConcurrentSkipListMap. The TreeMap was
slightly faster than a SkipList. The SkipList used slightly less RAM (by a few percent).

Cool, thanks for sharing.

bq. it would be able to support different concurrent update strategies.

I'm sure this would require more testing to see how concurrency pans out.

bq. If we have both data structures, we can do our first look-ups on Cache, which will normally
find the tablet, and that's a hash lookup, which is significantly faster. So that would be
a nice benefit.

Using two different data structures to maintain a single cache just comes across as a smell
to me. I'm sorry, I know this is coming across as me harping on this approach. Because the
locator is guaranteed to be used concurrently, this makes me worry even more about correctly
keeping state in two different places over one.

In a perfect world, the Locator impl is just holding a single copy of the data in the metadata
table. While the Guava Cache interface is nice to use, is it going to be ultimately a more
confusing implementation that have a single data structure to cache the metadata table?

> Add LRU mechanism to address memory concerns and possibly pre-fetch Tablet Locations.
> -------------------------------------------------------------------------------------
>                 Key: ACCUMULO-3982
>                 URL:
>             Project: Accumulo
>          Issue Type: New Feature
>          Components: client, tserver
>    Affects Versions: 1.6.3, 1.7.0, 1.8.0
>            Reporter: marco polo
>            Assignee: Eric Newton
>             Fix For: 1.8.0
> On larger systems, especially those of which may split frequently, I've seen churn in
the TabletLocator caches. Could we possibly pre-fetch entries for a table so that we don't
constantly access the metadata table for tablet locations. 
> Additionally, instead of clearing the cache in the case of a miss or failure, could we
simply update those entries in the cache? If we do this we may minimize cache misses overall
and the need to scan the meta data table. If need be, we could have a threshold before clearing
the cache entirely if people see the need for that.
> It would also be helpful to add an LRU cache ( or maybe even priority based LRU cache
). In ACCUMULO-3549, we added a cache clear to TabletServer to avoid overwhelming memory.
An LRU cache might address this concern and make the clear cache call unnecessary.

This message was sent by Atlassian JIRA

View raw message