cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Antti Nissinen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10496) Make DTCS split partitions based on time during compaction
Date Fri, 09 Oct 2015 12:41:27 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950316#comment-14950316
] 

Antti Nissinen commented on CASSANDRA-10496:
--------------------------------------------

Hello [~krummas], just in case if you have not run into this one [CASSANDRA-10306|https://issues.apache.org/jira/browse/CASSANDRA-10306].
Splitting of  SSTables in time wise would bring also a nice possibilities to clean data from
the column family effectively and based on the varying needs. TTL is a working solution if
you know the retention time for data before hand and also the amount of data collected. However,
at least in process industry the needs might change fairly quickly and in unpredictable manner.

> Make DTCS split partitions based on time during compaction
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-10496
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10496
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Marcus Eriksson
>              Labels: dtcs
>             Fix For: 3.x
>
>
> To avoid getting old data in new time windows with DTCS (or related, like [TWCS|CASSANDRA-9666]),
we need to split out old data into its own sstable during compaction.
> My initial idea is to just create two sstables, when we create the compaction task we
state the start and end times for the window, and any data older than the window will be put
in its own sstable.
> By creating a single sstable with old data, we will incrementally get the windows correct
- say we have an sstable with these timestamps:
> {{[100, 99, 98, 97, 75, 50, 10]}}
> and we are compacting in window {{[100, 80]}} - we would create two sstables:
> {{[100, 99, 98, 97]}}, {{[75, 50, 10]}}, and the first window is now 'correct'. The next
compaction would compact in window {{[80, 60]}} and create sstables {{[75]}}, {{[50, 10]}}
etc.
> We will probably also want to base the windows on the newest data in the sstables so
that we actually have older data than the window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message