Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Tue, 1 Sep 2015 20:15:46 +0000 (UTC)
From: "Eric Newton (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12860698.1441031669000.226605.1441138546059@Atlassian.JIRA>
In-Reply-To: <JIRA.12860698.1441031669000@Atlassian.JIRA>
References: <JIRA.12860698.1441031669000@Atlassian.JIRA>
 <JIRA.12860698.1441031669995@arcas>
Subject: [jira] [Commented] (ACCUMULO-3982) Add LRU mechanism to address
 memory concerns and possibly pre-fetch Tablet Locations.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726099#comment-14726099 ] 

Eric Newton commented on ACCUMULO-3982:
---------------------------------------

Re-implementing caches also has a bad smell.

LRU caches generally keep two data structures. So, under the covers, the Guava cache (... ok, inventing, since I didn't actually read the source...) probably has a hash table for fast look-ups and a priority queue to quickly compute what elements need to be removed. Having two references is required to look at the data from two perspectives.  We would be adding a third: Range-based look-ups. It is not ideal, since there is considerable overlap with the hash table. But it is less error prone than writing our own cache, documenting the implementation, and allowing it to be configurable.

[~phrocker] is experimenting on a very large deployed cluster that just removes the hourly invalidation. I imagine this is the implementation we will use and all of this cache optimization will be dropped. I'll report back on his experience.

> Add LRU mechanism to address memory concerns and possibly pre-fetch Tablet Locations.
> -------------------------------------------------------------------------------------
>
>                 Key: ACCUMULO-3982
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3982
>             Project: Accumulo
>          Issue Type: New Feature
>          Components: client, tserver
>    Affects Versions: 1.6.3, 1.7.0, 1.8.0
>            Reporter: marco polo
>            Assignee: Eric Newton
>             Fix For: 1.8.0
>
>
> On larger systems, especially those of which may split frequently, I've seen churn in the TabletLocator caches. Could we possibly pre-fetch entries for a table so that we don't constantly access the metadata table for tablet locations. 
> Additionally, instead of clearing the cache in the case of a miss or failure, could we simply update those entries in the cache? If we do this we may minimize cache misses overall and the need to scan the meta data table. If need be, we could have a threshold before clearing the cache entirely if people see the need for that.
> It would also be helpful to add an LRU cache ( or maybe even priority based LRU cache ). In ACCUMULO-3549, we added a cache clear to TabletServer to avoid overwhelming memory. An LRU cache might address this concern and make the clear cache call unnecessary.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)