Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 20 Jan 2015 09:32:34 +0000 (UTC)
From: "William Saar (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12765749.1420718588000.122294.1421746354693@Atlassian.JIRA>
In-Reply-To: <JIRA.12765749.1420718588000@Atlassian.JIRA>
References: <JIRA.12765749.1420718588000@Atlassian.JIRA>
 <JIRA.12765749.1420718588218@arcas>
Subject: [jira] [Updated] (CASSANDRA-8580) AssertionErrors after activating
 unchecked_tombstone_compaction with leveled compaction
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/CASSANDRA-8580?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

William Saar updated CASSANDRA-8580:
------------------------------------
    Attachment: system_omit_cold_reads_at_approx_0830.log

Attached log from one of the nodes. The exception starts occurring at about=
 08.30, shortly after the cold_reads_to_omit property was updated.

The system started clean and had some load from 9 PM to 1AM. We did the cha=
nge as some nodes had lots of pending compactions but no compactions runnin=
g. We ended up with these exceptions and one node without exceptions that h=
ad hundreds of SSTables of approx ~380 MB, but no pending compactions.

> AssertionErrors after activating unchecked_tombstone_compaction with leve=
led compaction
> -------------------------------------------------------------------------=
--------------
>
>                 Key: CASSANDRA-8580
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8580
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Bj=C3=B6rn Hachmann
>            Assignee: Benedict
>             Fix For: 2.1.3
>
>         Attachments: system.log, system_omit_cold_reads_at_approx_0830.lo=
g
>
>
> During our upgrade of Cassandra from version 2.0.7 to 2.1.2 we experience=
d a serious problem regarding the setting unchecked_tombstone_compaction in=
 combination with leveled compaction strategy.
> In order to prevent tombstone-threshold-warnings we activated the setting=
 for a specific table after the upgrade. Some time after that we observed n=
ew errors in our log files:
> {code}
> INFO  [CompactionExecutor:184] 2014-12-11 12:36:06,597 CompactionTask.jav=
a:136 - Compacting [SSTableReader(path=3D'/data/cassandra/data/system/compa=
ctions_in_progress/system-compactions_in_progress-ka-1848-Data.db'), SSTabl=
eReader(path=3D'/
> data/cassandra/data/system/compactions_in_progress/system-compactions_in_=
progress-ka-1847-Data.db'), SSTableReader(path=3D'/data/cassandra/data/syst=
em/compactions_in_progress/system-compactions_in_progress-ka-1845-Data.db')=
, SSTableReader
> (path=3D'/data/cassandra/data/system/compactions_in_progress/system-compa=
ctions_in_progress-ka-1846-Data.db')]
> ERROR [CompactionExecutor:183] 2014-12-11 12:36:06,613 CassandraDaemon.ja=
va:153 - Exception in thread Thread[CompactionExecutor:183,1,main]
> java.lang.AssertionError: /data/cassandra/data/metrigo_prod/new_user_data=
/metrigo_prod-new_user_data-tmplink-ka-705732-Data.db
>         at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKe=
yCount(SSTableReader.java:243) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(Comp=
actionTask.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(Dis=
kAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable=
.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInter=
nal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.exec=
ute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.CompactionManager$Backgroun=
dCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.ja=
r:2.1.2]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.=
java:471) ~[na:1.7.0_45]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:=
1.7.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx=
ecutor.java:1145) ~[na:1.7.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:615) [na:1.7.0_45]
>         at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>         {code}
> Obviously that error aborted the compaction and after some time the numbe=
r of pending compactions became very high on every node. Of course, this in=
 turn had a negative impact on several other metrics.
> After reverting the setting we had to restart all nodes. After that compa=
ctions could finish again and the pending compactions could be worked off.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)