cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Stevens <migh...@gmail.com>
Subject Re: Compaction Filter in Cassandra
Date Wed, 16 Mar 2016 01:36:30 GMT
We have been working on filtering compaction for a month or so (though we
call it deleting compaction, its implementation is as a filtering
compaction strategy).  The feature is nearing completion, and we have used
it successfully in a limited production capacity against DSE 4.8 series.

Our use case is that our records are written anywhere between a month, up
to several years before they are scheduled for deletion.  Tombstones are
too expensive, as we have tables with hundreds of billions of rows.  In
addition, traditional TTLs don't work for us because our customers are
permitted to change their retention policy such that already-written
records should not be deleted if they increase their retention after the
record was written (or vice versa).

We can clean up data more cheaply and more quickly with filtered compaction
than with tombstones and traditional compaction.  Our implementation is a
wrapper compaction strategy for another underlying strategy, so that you
can have the characteristics of whichever strategy makes sense in terms of
managing your SSTables, while interceding and removing records during
compaction (including cleaning up secondary indexes) that otherwise would
have survived into the new SSTable.

We are hoping to contribute it back to the community, so if you'd be
interested in helping test it out, I'd love to hear from you.

On Sat, Mar 12, 2016 at 5:12 AM Marcus Eriksson <krummas@gmail.com> wrote:

> We don't have anything like that, do you have a specific use case in mind?
>
> Could you create a JIRA ticket and we can discuss there?
>
> /Marcus
>
> On Sat, Mar 12, 2016 at 7:05 AM, Dikang Gu <dikang85@gmail.com> wrote:
>
>> Hello there,
>>
>> RocksDB has the feature called "Compaction Filter" to allow application
>> to modify/delete a key-value during the background compaction.
>> https://github.com/facebook/rocksdb/blob/v4.1/include/rocksdb/options.h#L201-L226
>>
>> I'm wondering is there a plan/value to add this into C* as well? Or is
>> there already a similar thing in C*?
>>
>> Thanks
>>
>> --
>> Dikang
>>
>>
>

Mime
View raw message