cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
Date Mon, 18 Apr 2016 23:43:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246776#comment-15246776
] 

Paulo Motta commented on CASSANDRA-11548:
-----------------------------------------

Good catch! I think this makes sense and there could be a race where an sstable is removed
from the compacting set during anti-compaction, and meanwhile is removed from the {{DataTracker}}
by a concurrent compaction, and when {{cfs.getDataTracker().markCompactedSSTablesReplaced(successfullyAntiCompactedSSTables,
anticompactedSSTables, OperationType.ANTICOMPACTION);}} is called at the end of the anti-compaction,
the {{java.lang.AssertionError: Expecting new size of 95, got 96 while replacing XXX by XXX}}
is thrown, causing the sstable to be silently not replaced after compaction, but still picked
up by a subsequent repair and cause the {{received out of order AssertionError}} on CASSANDRA-9935.

On 2.2+ this is fixed by CASSANDRA-8568, that extended the transactional API to sstables so
no need to the apply the patch there.

I did some minor update on the patch (removed unnecessary map {{successfullyAntiCompactedSSTables}})
and added a unit test to reproduce the issue. CI tests will be available shortly below:

||2.1||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:2.1-11548]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11548-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11548-dtest/lastCompletedBuild/testReport/]|

[~krummas] could you have a quick look to double check this makes sense and we haven't missed
anything?

While this will fix the root cause of the issue, on CASSANDRA-9935 I will add a protection
to prevent the {{received out of order AssertionError}} when a bug like this causes an sstable
not to be removed correctly.

> Anticompaction not removing old sstables
> ----------------------------------------
>
>                 Key: CASSANDRA-11548
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 2.1.13
>            Reporter: Ruoran Wang
>         Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable : repairedSSTables)```
> I think the effect of those above change might cause the markCompactedSSTablesReplaced
fail on 
> DataTracker.java
> {noformat}
>            assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
>                 String.format("Expecting new size of %d, got %d while replacing %s by
%s in %s",
>                           newSSTablesSize, newSSTables.size() + newShadowed.size(), oldSSTables,
replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught, leaving
the oldsstables not removed. (Then this might cause row out of order error when doing incremental
repair if there are L1 un-repaired sstables.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message