cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Sun, 07 Jul 2013 05:39:50 GMT


Jonathan Ellis commented on CASSANDRA-5661:

Perhaps it will help if I recap:

# CRAR pooling + LCS OOMs us at relatively small amounts of data (20GB in Jeremiah's example)
# worst case pool size proportional to sstables * concurrent readers means that memory usage
increases linearly with disk size; increasing sstable size decreases it linearly, so we expect
100MB sstables [which is on the high side of what is reasonable IMO] to get us to ~4TB of
space managed which is good but not "problem solved" territory. 
# thus, we need to bound CRAR pooling and not just say "it's working as designed, go use larger
sstables or STCS" [although that is probably adequate for 1.2, so I am tagging this for 2.0]
# The most common operations in an object pool are the borrow/return.  Ben's multiway pool
optimizes for this, with particular attention to possible contention from multiple reader
# The multiway pool supports expiring objects after they have been idle for a given time,
as well as a total pool size.

> Discard pooled readers for cold data
> ------------------------------------
>                 Key: CASSANDRA-5661
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 1.2.7
>         Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message