cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Wed, 26 Jun 2013 23:18:21 GMT


Pavel Yaskevich commented on CASSANDRA-5661:

After thinking about this more as LCS produces similar (same) sized files there is actually
no benefit of caching [C]RAR instances. I think we should do caching only with STCS + expiring
as it introduces less memory overhead per file for [C]RAR.

I also want to share my worries about LCS using 5MB file size with mmap and no compression:
default vm.max_map_count is 65536 (maximum number of memory mappings per process) which is
around 322GB of data, we need to mention somewhere that if somebody wants to run with mmap
+ LCS they need to adjust that setting. Another thing that worries me is that performance
of mmap, munmap and similar + minor/major page faults would degrade logarithmically as memory
mappings are handled by red-black tree and with constant compaction we would be burning a
lot more cpu on tree balancing as dataset grows.
> Discard pooled readers for cold data
> ------------------------------------
>                 Key: CASSANDRA-5661
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 1.2.7
>         Attachments: DominatorTree.png, Histogram.png
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message