cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-4411) Assertion with LCS compaction
Date Fri, 13 Jul 2012 13:17:35 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-4411:
----------------------------------------

    Attachment: 0001-Add-debugging-info-for-LCS.txt

Unfortunately the log doesn't give us much to chew on.

I've actually be able to reproduce this once (using stress). Unfortunately, I hadn't added
more debugging yet and since I added more debug info I haven't been able to reproduce (despite
having retried from scratch like 3 times letting it run for multiple hours each time).

So I'm attaching a simple patch that adds more debugging. If you guys can try applying the
patch and see if you can reproduce. If so, the log file produced should be helpful. I'll note
that it is preferable to *not* turn DEBUG logging with this patch as this is not useful and
would only generate awfully large logs.

                
> Assertion with LCS compaction
> -----------------------------
>
>                 Key: CASSANDRA-4411
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4411
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>            Reporter: Anton Winter
>             Fix For: 1.1.3
>
>         Attachments: 0001-Add-debugging-info-for-LCS.txt, assertion.system.log, system.log
>
>
> As instructed in CASSANDRA-4321 I have raised this issue as a continuation of that issue
as it appears the problem still exists.
> I have repeatedly run sstablescrub across all my nodes after the 1.1.2 upgrade until
sstablescrub shows no errors.  The exceptions described in CASSANDRA-4321 do not occur as
frequently now but the integrity check still throws exceptions on a number of nodes.  Once
those exceptions occur compactionstats shows a large number of pending tasks with no progression
afterwards.
> {code}
> ERROR [CompactionExecutor:150] 2012-07-05 04:26:15,570 AbstractCassandraDaemon.java (line
134) Exception in thread Thread[CompactionExecutor:150,1,main]
> java.lang.AssertionError
>         at org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214)
>         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158)
>         at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:978)
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
>         at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>         at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message