Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F1445C2AF for ; Thu, 8 Jan 2015 15:41:33 +0000 (UTC) Received: (qmail 32009 invoked by uid 500); 8 Jan 2015 15:41:35 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 31963 invoked by uid 500); 8 Jan 2015 15:41:35 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 31951 invoked by uid 99); 8 Jan 2015 15:41:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jan 2015 15:41:35 +0000 Date: Thu, 8 Jan 2015 15:41:34 +0000 (UTC) From: "Philip Thompson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-8580) AssertionErrors after activating unchecked_tombstone_compaction with leveled compaction MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-8580?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8580: --------------------------------------- Description:=20 During our upgrade of Cassandra from version 2.0.7 to 2.1.2 we experienced = a serious problem regarding the setting unchecked_tombstone_compaction in c= ombination with leveled compaction strategy. In order to prevent tombstone-threshold-warnings we activated the setting f= or a specific table after the upgrade. Some time after that we observed new= errors in our log files: {code} INFO [CompactionExecutor:184] 2014-12-11 12:36:06,597 CompactionTask.java:= 136 - Compacting [SSTableReader(path=3D'/data/cassandra/data/system/compact= ions_in_progress/system-compactions_in_progress-ka-1848-Data.db'), SSTableR= eader(path=3D'/ data/cassandra/data/system/compactions_in_progress/system-compactions_in_pr= ogress-ka-1847-Data.db'), SSTableReader(path=3D'/data/cassandra/data/system= /compactions_in_progress/system-compactions_in_progress-ka-1845-Data.db'), = SSTableReader (path=3D'/data/cassandra/data/system/compactions_in_progress/system-compact= ions_in_progress-ka-1846-Data.db')] ERROR [CompactionExecutor:183] 2014-12-11 12:36:06,613 CassandraDaemon.java= :153 - Exception in thread Thread[CompactionExecutor:183,1,main] java.lang.AssertionError: /data/cassandra/data/metrigo_prod/new_user_data/m= etrigo_prod-new_user_data-tmplink-ka-705732-Data.db at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyC= ount(SSTableReader.java:243) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.runWith(Compac= tionTask.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskA= wareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:28) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.executeInterna= l(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execut= e(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundC= ompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:= 2.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) ~[na:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.= 7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) ~[na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] {code} Obviously that error aborted the compaction and after some time the number = of pending compactions became very high on every node. Of course, this in t= urn had a negative impact on several other metrics. After reverting the setting we had to restart all nodes. After that compact= ions could finish again and the pending compactions could be worked off. was: During our upgrade of Cassandra from version 2.0.7 to 2.1.2 we experienced = a serious problem regarding the setting unchecked_tombstone_compaction in c= ombination with leveled compaction strategy. In order to prevent tombstone-threshold-warnings we activated the setting f= or a specific table after the upgrade. Some time after that we observed new= errors in our log files: INFO [CompactionExecutor:184] 2014-12-11 12:36:06,597 CompactionTask.java:= 136 - Compacting [SSTableReader(path=3D'/data/cassandra/data/system/compact= ions_in_progress/system-compactions_in_progress-ka-1848-Data.db'), SSTableR= eader(path=3D'/ data/cassandra/data/system/compactions_in_progress/system-compactions_in_pr= ogress-ka-1847-Data.db'), SSTableReader(path=3D'/data/cassandra/data/system= /compactions_in_progress/system-compactions_in_progress-ka-1845-Data.db'), = SSTableReader (path=3D'/data/cassandra/data/system/compactions_in_progress/system-compact= ions_in_progress-ka-1846-Data.db')] ERROR [CompactionExecutor:183] 2014-12-11 12:36:06,613 CassandraDaemon.java= :153 - Exception in thread Thread[CompactionExecutor:183,1,main] java.lang.AssertionError: /data/cassandra/data/metrigo_prod/new_user_data/m= etrigo_prod-new_user_data-tmplink-ka-705732-Data.db at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyC= ount(SSTableReader.java:243) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.runWith(Compac= tionTask.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskA= wareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:28) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.executeInterna= l(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execut= e(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundC= ompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:= 2.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) ~[na:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.= 7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) ~[na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] =20 Obviously that error aborted the compaction and after some time the number = of pending compactions became very high on every node. Of course, this in t= urn had a negative impact on several other metrics. After reverting the setting we had to restart all nodes. After that compact= ions could finish again and the pending compactions could be worked off. > AssertionErrors after activating unchecked_tombstone_compaction with leve= led compaction > -------------------------------------------------------------------------= -------------- > > Key: CASSANDRA-8580 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8580 > Project: Cassandra > Issue Type: Bug > Reporter: Bj=C3=B6rn Hachmann > > During our upgrade of Cassandra from version 2.0.7 to 2.1.2 we experience= d a serious problem regarding the setting unchecked_tombstone_compaction in= combination with leveled compaction strategy. > In order to prevent tombstone-threshold-warnings we activated the setting= for a specific table after the upgrade. Some time after that we observed n= ew errors in our log files: > {code} > INFO [CompactionExecutor:184] 2014-12-11 12:36:06,597 CompactionTask.jav= a:136 - Compacting [SSTableReader(path=3D'/data/cassandra/data/system/compa= ctions_in_progress/system-compactions_in_progress-ka-1848-Data.db'), SSTabl= eReader(path=3D'/ > data/cassandra/data/system/compactions_in_progress/system-compactions_in_= progress-ka-1847-Data.db'), SSTableReader(path=3D'/data/cassandra/data/syst= em/compactions_in_progress/system-compactions_in_progress-ka-1845-Data.db')= , SSTableReader > (path=3D'/data/cassandra/data/system/compactions_in_progress/system-compa= ctions_in_progress-ka-1846-Data.db')] > ERROR [CompactionExecutor:183] 2014-12-11 12:36:06,613 CassandraDaemon.ja= va:153 - Exception in thread Thread[CompactionExecutor:183,1,main] > java.lang.AssertionError: /data/cassandra/data/metrigo_prod/new_user_data= /metrigo_prod-new_user_data-tmplink-ka-705732-Data.db > at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKe= yCount(SSTableReader.java:243) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.compaction.CompactionTask.runWith(Comp= actionTask.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(Dis= kAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.compaction.CompactionTask.executeInter= nal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.compaction.AbstractCompactionTask.exec= ute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.compaction.CompactionManager$Backgroun= dCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.ja= r:2.1.2] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:= 1.7.0_45] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) ~[na:1.7.0_45] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > {code} > Obviously that error aborted the compaction and after some time the numbe= r of pending compactions became very high on every node. Of course, this in= turn had a negative impact on several other metrics. > After reverting the setting we had to restart all nodes. After that compa= ctions could finish again and the pending compactions could be worked off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)