cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Manes (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Sat, 13 Jul 2013 04:17:49 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707652#comment-13707652
] 

Ben Manes commented on CASSANDRA-5661:
--------------------------------------

I rewrote the time-to-idle policy, so it should be faster when enabled. 

Details (if interested)
------------------------
For prototyping purposes, I previously used a secondary Guava Cache to track idle resources.
Unlike a cache's time-to-idle, which is reset when an entry is read, an object pool's concept
of idle time is when a resource resides unused and ready to be borrowed. The use of a secondary
Guava Cache meant that the resource had to be added and removed frequently, resulting in locking
on the hashtable segments and incurring other maintenance overhead.

In Guava's cache we observed that expiration policies mirrored maximum size policies, but
time based. Thus time-to-live is a FIFO queue and time-to-idle is an LRU queue. That let us
leverage the amortization technique in CLHM to be used for expiration with O(1) reorder costs.

The new implementation strips off the unnecessary work by maintaining a time ordered queue
that only supports adds and removals. For our definition of idle there is no need to reorder
so it is effectively a FIFO. A tryLock guards the policy operations, draining a queue of pending
operations if acquired. I decided to allow this to be proactively drained whenever possible,
though if we see a need then we can buffer the operations for longer like the caches do.
                
> Discard pooled readers for cold data
> ------------------------------------
>
>                 Key: CASSANDRA-5661
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5661
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 2.0
>
>         Attachments: CASSANDRA-5661-multiway-per-sstable.patch, CASSANDRA-5661.patch,
DominatorTree.png, Histogram.png
>
>
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message