Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 737D7200BE2 for ; Thu, 1 Dec 2016 01:48:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 7212D160B19; Thu, 1 Dec 2016 00:48:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9939B160B13 for ; Thu, 1 Dec 2016 01:47:59 +0100 (CET) Received: (qmail 43937 invoked by uid 500); 1 Dec 2016 00:47:58 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 43924 invoked by uid 99); 1 Dec 2016 00:47:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Dec 2016 00:47:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 934032C2A67 for ; Thu, 1 Dec 2016 00:47:58 +0000 (UTC) Date: Thu, 1 Dec 2016 00:47:58 +0000 (UTC) From: "Simon Zhou (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 01 Dec 2016 00:48:00 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710374#comment-15710374 ] Simon Zhou commented on CASSANDRA-12203: ---------------------------------------- Also saw exactly the same issue after upgrading from 2.2.5 to 3.0.10. > AssertionError on compaction after upgrade (2.1.9 -> 3.7) > --------------------------------------------------------- > > Key: CASSANDRA-12203 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12203 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.7 (upgrade from 2.1.9) > Java version "1.8.0_91" > Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64) > Reporter: Roman S. Borschel > Fix For: 3.x > > > After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family (using SizeTieredCompaction) repeatedly and continuously failed compaction (and thus also repair) across the cluster, with all nodes producing the following errors in the logs: > {noformat} > 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread Thread[CompactionExecutor:3,1,main] > 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null > 2016-07-14T09:29:47.96859 |srv=cassandra| at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96860 |srv=cassandra| at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96860 |srv=cassandra| at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96860 |srv=cassandra| at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96861 |srv=cassandra| at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96861 |srv=cassandra| at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96862 |srv=cassandra| at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96862 |srv=cassandra| at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96863 |srv=cassandra| at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96863 |srv=cassandra| at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96864 |srv=cassandra| at org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96864 |srv=cassandra| at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96865 |srv=cassandra| at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96865 |srv=cassandra| at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96866 |srv=cassandra| at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96866 |srv=cassandra| at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96867 |srv=cassandra| at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96867 |srv=cassandra| at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96867 |srv=cassandra| at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96868 |srv=cassandra| at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96868 |srv=cassandra| at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96869 |srv=cassandra| at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-3.7.jar:3.7] > 2016-07-14T09:29:47.96870 |srv=cassandra| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91] > 2016-07-14T09:29:47.96870 |srv=cassandra| at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91] > 2016-07-14T09:29:47.96870 |srv=cassandra| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91] > 2016-07-14T09:29:47.96871 |srv=cassandra| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] > 2016-07-14T09:29:47.96871 |srv=cassandra| at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > {noformat} > Only 1 out of 14 tables was affected by this problem after the upgrade. > The schema of the affected table looks like this: > {noformat} > create columnfamily if not exists cassandra12203 > ( field1 uuid > , field2 ascii > , field3 int > , field4 text > , primary key (field1, field2, field3) > ); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)