cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marco Cadetg (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
Date Fri, 26 Feb 2016 14:09:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169058#comment-15169058
] 

Marco Cadetg commented on CASSANDRA-11172:
------------------------------------------

Here is the exception what happens prior the log spamming starts
{code}
ERROR [CompactionExecutor:5] 2016-02-26 14:05:26,622 CassandraDaemon.java:195 - Exception
in thread Thread[CompactionExecutor:5,1,main]
java.lang.AssertionError: null
        at org.apache.cassandra.db.rows.BufferCell.<init>(BufferCell.java:49) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:88) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:83) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BufferCell.purge(BufferCell.java:175) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.ComplexColumnData.lambda$purge$107(ComplexColumnData.java:165)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.ComplexColumnData$$Lambda$126/379481423.apply(Unknown
Source) ~[na:na]
        at org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.ComplexColumnData.transformAndFilter(ComplexColumnData.java:170)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:165)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:43)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BTreeRow.lambda$purge$102(BTreeRow.java:333) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BTreeRow$$Lambda$125/1572342504.apply(Unknown Source)
~[na:na]
        at org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:338) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:333) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:116) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:118)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:74)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:132)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
~[apache-cassandra-3.3.0.jar:3.3.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}

> Infinite loop bug adding high-level SSTableReader in compaction
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-11172
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: DSE 4.x / Cassandra 2.1.11.969
>            Reporter: Jeff Ferland
>            Assignee: Marcus Eriksson
>             Fix For: 2.1.14, 2.2.6, 3.0.4, 3.4
>
>         Attachments: beep.txt, tpstats.txt, tpstats_compaction.txt, trapped_in_compaction.txt,
trapped_in_compaction_mixed.txt
>
>
> Observed that after a large repair on LCS that sometimes the system will enter an infinite
loop with vast amounts of logs lines recording, "Adding high-level (L${LEVEL}) SSTableReader(path='${TABLE}')
to candidates"
> This results in an outage of the node and eventual crashing. The log spam quickly rotates
out possibly useful earlier debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message