cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-5342) ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables are removed
Date Thu, 20 Jun 2013 08:56:21 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcus Eriksson updated CASSANDRA-5342:
---------------------------------------

    Attachment: 0001-CASSANDRA-5342-wip.patch

removes ancestors from SSTableMetadata and instead makes deserialize(..) return a Pair<SSTM,
Set<Integer>> so that the caller can decide if the ancestors are needed. This allows
us to keep SSTM as immutable as possible.

this forces us to re-deserialize the metadata when trying to figure out ancestors during compaction

opted not to mutate the ancestors on-disk since it makes my skin crawl
                
> ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables
are removed
> ----------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5342
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5342
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.3
>            Reporter: Wei Zhu
>            Assignee: Marcus Eriksson
>             Fix For: 1.2.7
>
>         Attachments: 0001-CASSANDRA-5342-wip.patch, Screen Shot 2013-03-13 at 12.05.08
PM.png
>
>
> We are using LCS and have total of 38000 SSTables for one CF. During LCS, there could
be over a thousand SSTable involved. All those SSTable IDs are stored in ancestors field of
SSTableMetatdata for the new table. In our case, it consumes more than 1G of heap memory for
those field. Put it in perspective, the ancestors consume 2 - 3 times more memory than bloomfilter
(fp = 0.1 by default) in LCS. 
> We should remove those ancestors from SSTableMetadata after the compaction is finished
and the old SSTable is removed. It  might be a big deal for Sized Compaction since there are
small number of SSTable involved. But it consumes a lot of memory for LCS. 
> At least, we shouldn't load those ancestors to the memory during startup if the files
are removed. 
> I would love to contribute and provide patch. Please let me know how to start. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message