cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J. Ryan Earl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6086) Node refuses to start with exception in ColumnFamilyStore.removeUnfinishedCompactionLeftovers when find that some to be removed files are already removed
Date Fri, 08 Nov 2013 22:26:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817757#comment-13817757
] 

J. Ryan Earl commented on CASSANDRA-6086:
-----------------------------------------

Hi, we are able to consistently reproduce this issue:
{noformat}
ERROR 23:14:06,001 Exception encountered during startup
java.lang.IllegalStateException: Unfinished compactions reference missing sstables. This should
never happen since compactions are marked finished before we start removing the old sstables.
       at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:489)
       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:264)
       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
java.lang.IllegalStateException: Unfinished compactions reference missing sstables. This should
never happen since compactions are marked finished before we start removing the old sstables.
       at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:489)
       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:264)
       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
Exception encountered during startup: Unfinished compactions reference missing sstables. This
should never happen since compactions are marked finished before we start removing the old
sstables.
{noformat}

Here are the two ways in which we have found to reproduce this issue:
# Buffer a large amount of CL.LOCAL_QUORUM writes into a Cassandra CF, then drop the CF while
the writes are still buffered.
# Buffer a large amount of CL.LOCAL_QUORUM writes into a Cassandra CF, then restart some nodes
while before the buffer has finished draining.

> Node refuses to start with exception in ColumnFamilyStore.removeUnfinishedCompactionLeftovers
when find that some to be removed files are already removed
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6086
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6086
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Oleg Anastasyev
>            Assignee: Oleg Anastasyev
>             Fix For: 2.0.3
>
>         Attachments: 6086-v2.txt, removeUnfinishedCompactionLeftovers.txt
>
>
> Node refuses to start with
> {code}
> Caused by: java.lang.IllegalStateException: Unfinished compactions reference missing
sstables. This should never happen since compactions are marked finished before we start removing
the old sstables.
>       at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:544)
>       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:262)
> {code}
> IMO, there is no reason to refuse to start discivering files that must be removed are
already removed. It looks like pure bug diagnostic code and mean nothing to operator (nor
he can do anything about this).
> Replaced throw of excepion with dump of diagnostic warning and continue startup.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message