cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12706) Exception supposedly after ttl expires and compaction occurs
Date Mon, 26 Sep 2016 09:15:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522538#comment-15522538
] 

Stefania commented on CASSANDRA-12706:
--------------------------------------

I think this very simple [patch|https://github.com/stef1927/cassandra/commits/12706-3.0] should
fix it but, could you share your schema and the exact CQL query syntax so that we can try
to reproduce the problem? 

This should be reproducible with the same slice type, which depends on the CQL, and when the
tombstones exceed the warning threshold - because of the code in {{ReadCommand.MetricsRecorder.onClose}}:

{code}
boolean warnTombstones = tombstones > warningThreshold && respectTombstoneThresholds;
if (warnTombstones)
{
    String msg = String.format("Read %d live rows and %d tombstone cells for query %1.512s
(see tombstone_warn_threshold)", liveRows, tombstones, ReadCommand.this.toCQLString());
    ClientWarn.instance.warn(msg);
    logger.warn(msg);
}
{code}

The NPE is ultimately caused by the call to {{toCQLString()}}. {{respectTombstoneThresholds}}
is always true for non system tables, whilst {{warningThreshold}} comes from the tombstone
warning threshold in cassandra.yaml.

> Exception supposedly after ttl expires and compaction occurs
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-12706
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12706
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: Linux amd64 Amazon default AMI,
> Java 1.8.0_122 oracle jdk
>            Reporter: Nikhil Sharma
>            Priority: Critical
>
> After TTL (1 week) expires and compaction occurs. We start seeing:
> WARN  [SharedPool-Worker-1] 2016-09-26 07:35:17,420 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.NullPointerException: null
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> WARN  [SharedPool-Worker-3] 2016-09-26 07:35:31,247 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-3,5,main]: {}
> java.lang.RuntimeException: java.lang.NullPointerException
>        	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: java.lang.NullPointerException: null
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	... 5 common frames omitted
> This results in queries failing on those tables (not always though, at first). The failures
keep increasing, until total failure after a few days.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message