cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Wed, 26 Jun 2013 23:18:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13694344#comment-13694344
] 

Pavel Yaskevich commented on CASSANDRA-5661:
--------------------------------------------

After thinking about this more as LCS produces similar (same) sized files there is actually
no benefit of caching [C]RAR instances. I think we should do caching only with STCS + expiring
as it introduces less memory overhead per file for [C]RAR.

I also want to share my worries about LCS using 5MB file size with mmap and no compression:
default vm.max_map_count is 65536 (maximum number of memory mappings per process) which is
around 322GB of data, we need to mention somewhere that if somebody wants to run with mmap
+ LCS they need to adjust that setting. Another thing that worries me is that performance
of mmap, munmap and similar + minor/major page faults would degrade logarithmically as memory
mappings are handled by red-black tree and with constant compaction we would be burning a
lot more cpu on tree balancing as dataset grows.
                
> Discard pooled readers for cold data
> ------------------------------------
>
>                 Key: CASSANDRA-5661
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5661
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 1.2.7
>
>         Attachments: DominatorTree.png, Histogram.png
>
>
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message