cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Cruz <>
Subject Re: Repair completes successfully but data is still inconsistent
Date Wed, 19 Nov 2014 13:18:25 GMT
On 19 Nov 2014, at 11:37, André Cruz <> wrote:
> All the nodes were restarted on 21-23 October, for the upgrade (1.2.16 -> 1.2.19)
I mentioned. The delete happened after. I should also point out that we were experiencing
problems related to CASSANDRA-4206 and CASSANDRA-7808.

Another possible cause are these exceptions I found in the log as the nodes were shutdown
and brought up with the new version:

INFO [RMI TCP Connection(270364)-] 2014-10-21 15:04:00,867
(line 939) DRAINED
ERROR [CompactionExecutor:15173] 2014-10-21 15:04:01,923 (line 191) Exception
in thread Thread[CompactionExecutor:15173,1,main]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1dad30c0
rejected from org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@555b9c78[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14052]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Unknown Source)
        at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(
        at org.apache.cassandra.db.DataTracker.postReplace(
        at org.apache.cassandra.db.DataTracker.replace(
        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(
        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(
        at org.apache.cassandra.db.compaction.CompactionTask.replaceCompactedSSTables(
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(
        at org.apache.cassandra.db.compaction.CompactionManager$
        at java.util.concurrent.Executors$ Source)
        at Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$ Source)
        at Source)

Each node has 4-9 of these exceptions as it is going down after being drained. It seems Cassandra
was trying to delete an sstable. Can this be related?

Best regards,
André Cruz
View raw message