cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-7803) When compaction is interrupted, it leaves locked, undeletable files
Date Thu, 21 Aug 2014 07:30:40 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-7803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcus Eriksson updated CASSANDRA-7803:
---------------------------------------

    Attachment: 0001-dont-try-to-remove-fake-FINAL-files.patch

When opening the compaction result early we fake the descriptor to be FINAL, then, when we
abort the compaction we try to remove the FINAL file which does not exist. Patch checks if
the sstable is opened early and removes the TMPLINK files instead.

> When compaction is interrupted, it leaves locked, undeletable files
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-7803
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7803
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: RedHat, xfs4, JNA enabled, JBOD,
>            Reporter: Scooletz
>            Assignee: Marcus Eriksson
>              Labels: compaction
>             Fix For: 2.1.0
>
>         Attachments: 0001-dont-try-to-remove-fake-FINAL-files.patch
>
>
> During tests of new features of 2.1 like: 
> - incremental repairs 
> - leveled compaction
> I interrupted a compaction, which left the following ERROR in the _system.log_
> {quote}
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction interrupted:
Compaction@152e6e70-1975-11e4-ba09-61f0d75c60c6(xx, xxx, 378505918/1993581634)bytes
> 	at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:174)
> 	at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74)
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> 	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:235)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_09]
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.7.0_09]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) ~[na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> {quote}
> Right after that, a cascade of reoccurring errors was being emitted till the restart:
> {quote}
> ERROR [NonPeriodicTasks:1] 2014-08-19 13:38:41,258 SSTableDeletingTask.java:81 - Unable
to delete /grid/data04/cassandra/data/xx/xxx-152e6e70197511e4ba0961f0d75c60c6/xx-xxx-ka-55058-Data.db
(it will be removed on server restart; we'll also retry after GC)
> {quote}
> which made this node blinking (noted from the other nodes gossiper log entries).
> After restart, the node is healthy and fully operational.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message