cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5661) Discard pooled readers for cold data
Date Thu, 27 Jun 2013 19:30:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13694977#comment-13694977
] 

Pavel Yaskevich commented on CASSANDRA-5661:
--------------------------------------------

Well it depends on how do you define equal, having 1TB of data would definitely require bigger
heap and physical memory configuration. 

Let's calculate (where each file have one buffer in memory at all times):

5MB   files (each 79KB decompression buffer) for 1GB of such files in memory would be: 204
(num files in 1GB) * 79KB = *16MB* buffers
128MB files (-//-) require 25.5 times less buffers per 1GB than 5MB files: 16MB (buffers per
1GB in case of 5MB files) / 25.5 = *643KB* buffers

So for 1TB with 5MB files we need 1024 * 16MB = *16GB* of heap and for 128MB files it's 25.5
times less = *643MB*, if each of the files is going to have at least 8 caches items in the
same time with 128MB files we are going to have around 5GB of heap but I do think this scenario
is a worst case, normal mode would be 2-3GB. If you go with 14-16GB heap and 1TB of data,
2GB of cache is the least of your problems as it's around 10% of total heap size which is
still good trade-off to allocation rate if those buffers are allocated per call.
                
> Discard pooled readers for cold data
> ------------------------------------
>
>                 Key: CASSANDRA-5661
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5661
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.1
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 1.2.7
>
>         Attachments: DominatorTree.png, Histogram.png
>
>
> Reader pooling was introduced in CASSANDRA-4942 but pooled RandomAccessReaders are never
cleaned up until the SSTableReader is closed.  So memory use is "the worst case simultaneous
RAR we had open for this file, forever."
> We should introduce a global limit on how much memory to use for RAR, and evict old ones.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message