cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Sun, 07 Jul 2013 05:01:51 GMT


Pavel Yaskevich commented on CASSANDRA-5661:

bq. Sure, and this is supported, but this is not a case of contention which was the concern
I raised with Ben.

What contention are we talking? If that is memory pressure, then expiring items in bulk is
good and leaves room for new files. In case of LCS instances mostly are going to be invalidated
by compaction vs access time expiry, STCS observes the same effect but better in terms of
memory usage as smaller files are going to be compacted into bigger one so each compaction
reduces number of instances and buffers. 

I think the original problem in here was that with default settings LCS eats a lot of memory
because files are too small and even if minor portion of dataset is read memory, overhead
for caching is still unacceptable, this is why I was talking about adding global memory cap
and as a good bonus - expiry of unused instances, because if file is expired by timer, it
means that reads have turned away from it or it's read in bursts so deallocating all of the
cached instances is the way to go, where with LRU we would only replace when new instances
are returned to the pull (as we don't pre-allocate) which could create a problem when there
is a burst of requests to the same file after long internal of inactivity. But even after
so many comments that we had here and people ([~tjake]) reporting that even current setup
works for them on bigger files, it still looks like the everybody is trying to solve different
> Discard pooled readers for cold data
> ------------------------------------
>                 Key: CASSANDRA-5661
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 1.2.7
>         Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message