cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1902) Migrate cached pages during compaction
Date Fri, 18 Feb 2011 00:45:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996177#comment-12996177
] 

Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------

... actually the right limit is system memory, since we don't care how much is UNcached.

I'm estimating an upper bound of about 3GB -- say 128GB for a ludicrously large cache = 32M
entries, times 96B for each Pair.

On the one hand if you have that much memory you can afford to use an extra 3GB of heap during
compaction, but memory pressure that occurs only infrequently can be an unpleasant surprise
as we saw for CASSANDRA-2175.

If there's a sane way to do this w/o materializing the entire list let's do that.

> Migrate cached pages during compaction 
> ---------------------------------------
>
>                 Key: CASSANDRA-1902
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.7.1
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>             Fix For: 0.7.3
>
>         Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
>   Original Estimate: 32h
>          Time Spent: 24h
>  Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted
CF during the compaction process.  
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix
mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually
in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and
make sure the subsequent pages in the new compacted SSTable are kept in the page cache for
these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message