cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Goffinet (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1902) Migrate cached pages during compaction
Date Thu, 17 Feb 2011 23:09:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996145#comment-12996145
] 

Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------

I'd like to propose a change that might work much better for longer compactions: We write
out a new SSTable and using posix_fadvice (like we do today), and make sure nothing is in
cache. After the file is done being written, we call getCachedPages() across all sstables
used in compaction and compute which pages are hot AFTER compaction is complete. This would
allow us to to then sweep through the new SSTable written and mark pages that were hot. If
we do the process while file is being written and we have a compaction that might take an
hour, by the time it's done, the cache could churn.

> Migrate cached pages during compaction 
> ---------------------------------------
>
>                 Key: CASSANDRA-1902
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.7.1
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>             Fix For: 0.7.3
>
>         Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
>   Original Estimate: 32h
>          Time Spent: 24h
>  Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted
CF during the compaction process.  
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix
mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually
in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and
make sure the subsequent pages in the new compacted SSTable are kept in the page cache for
these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message