cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Sun, 07 Jul 2013 05:59:51 GMT


Pavel Yaskevich commented on CASSANDRA-5661:

bq. worst case pool size proportional to sstables * concurrent readers means that memory usage
increases linearly with disk size; increasing sstable size decreases it linearly, so we expect
100MB sstables [which is on the high side of what is reasonable IMO] to get us to ~4TB of
space managed which is good but not "problem solved" territory.

Let's clarify that it's a really-really worse case which means that your reads touch whole
dataset all the time and it's a good show case for side affect of having fixed size files
of small size so even if you don't do caching it would create allocation rate proportional
to the dataset size as well as increased syscall rate to open/seek/close those files so it's
frequent GC/FullGC vs OOM in here (when run without memory cap).

bq. The most common operations in an object pool are the borrow/return. Ben's multiway pool
optimizes for this, with particular attention to possible contention from multiple reader

If I understand correctly by borrow you mean allocate object in the pool and give it way to
the caller, this is not how I would like it to behave instead we can allocate object without
assigning to the queue when it runs short (as all of the objects are short lived) and then
decide if we want it back when caller it done with it, disadvantages of allocating to the
pool I have already described in my previous comments.

Bottom line for me being, I'm tired of arguing about this so I will let mighty people to decide
what they see fit as in all other cases (e.g. fadvic'ing whole file on reads, preheating page
cache etc.). 

> Discard pooled readers for cold data
> ------------------------------------
>                 Key: CASSANDRA-5661
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 1.2.7
>         Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message