cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: User Defined Compaction Issue
Date Tue, 26 Sep 2017 12:40:05 GMT
Write row A, flush into sstable 1
Delete row A, flush the tombstone into sstable 2

The tombstone in sstable 2 can’t be removed until row A in sstable 1 gets removed. If you
just keep recompacting sstable 2 by itself, the row in sstable A remains on disk.



-- 
Jeff Jirsa


> On Sep 26, 2017, at 2:01 AM, shalom sagges <shalomsagges@gmail.com> wrote:
> 
> Thanks Jeff!
> 
> I'll try that. 
> I'm not sure I understand how the tombstones are covering data in another file. Do you
have a small example perhaps?
> 
> Thanks again!
> 
>> On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>> The problem is likely that your sstables overlap - your 91% droppable tombstones
are probably covering data in another file. Until that data is removed, those tombstones cant
be dropped.
>> 
>> This is why a full major compaction often works better for simply reclaiming disk
space (though it takes a lot more disk as it runs and has other drawbacks) - it joins the
shadowed data with the tombstones into a single file. You can sorta fake that by using user-defined
compaction with multiple sstables to try to compact away the shadowed data - pick 2 or more
-Data files and compact them together. Works best if you can be sure which data is overlapping,
but short of that you'll probably want to pick data with approximately the same (or older)
calendar timestamps.
>> 
>> 
>> 
>>> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges <shalomsagges@gmail.com>
wrote:
>>> Hi Everyone, 
>>> 
>>> I'm running into an issue I can't seem to Solve. 
>>> 
>>> I execute force compaction in order to reclaim back storage. 
>>> Everything was working fine for a time, but after a while I found that tombstones
aren't being removed any longer. 
>>> 
>>> For example, I've compacted the following SSTable:
>>> 21G Sep 19 10:18 file-jb-69103-Data.db
>>> Estimated droppable tombstones: 0.9115601492568154
>>> 
>>> I ran it with jmxterm and the compaction started and completed successfully.

>>>  $>run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction
file-jb-69103-Data.db
>>>  #calling operation forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=CompactionManager
>>>  #operation returns: 
>>> null
>>> 
>>> 
>>> A new SStable was created, but no tombstones were removed. 
>>>  21G Sep 25 12:16 file-jb-71319-Data.db
>>>  Estimated droppable tombstones: 0.9115601492568154
>>> 
>>> I've tried running it from jconsole as well, but the outcome was the same. 
>>> 
>>> Is anyone familiar with this issue? 
>>> Btw, this cluster is on version 2.0.14 . 
>>> 
>>> 
>>> Thanks!
>>> Shalom
>>> 
>> 
> 

Mime
View raw message