cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikhil Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-12706) Exception supposedly after ttl expires and compaction occurs
Date Mon, 26 Sep 2016 08:32:20 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nikhil Sharma updated CASSANDRA-12706:
--------------------------------------
    Description: 
After TTL (1 week) expires and compaction occurs. We start seeing:

WARN  [SharedPool-Worker-1] 2016-09-26 07:35:17,420 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
       	at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
       	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9]
       	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
WARN  [SharedPool-Worker-3] 2016-09-26 07:35:31,247 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-3,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
       	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
       	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9]
       	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.lang.NullPointerException: null
       	at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76) ~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.0.9.jar:3.0.9]
       	... 5 common frames omitted

This results in queries failing on those tables (not always though, at first). The failures
keep increasing, until total failure after a few days.

  was:
After TTL (1 week) expires and compaction occurs. We start seeing:

WARN [SharedPool-Worker-1] 2016-09-26 04:07:19,849 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.NullPointerException:
null at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.9.jar:3.0.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.9.jar:3.0.9] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
WARN [SharedPool-Worker-2] 2016-09-26 04:07:20,639 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {} java.lang.RuntimeException:
java.lang.NullPointerException at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
~[apache-cassandra-3.0.9.jar:3.0.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.9.jar:3.0.9] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.lang.NullPointerException: null at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.0.9.jar:3.0.9] ... 5 common frames omitted

This results in queries failing on those tables (not always though, at first). The failures
keep increasing, until total failure after a few days.


> Exception supposedly after ttl expires and compaction occurs
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-12706
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12706
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: Linux amd64 Amazon default AMI,
> Java 1.8.0_122 oracle jdk
>            Reporter: Nikhil Sharma
>            Priority: Critical
>
> After TTL (1 week) expires and compaction occurs. We start seeing:
> WARN  [SharedPool-Worker-1] 2016-09-26 07:35:17,420 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.NullPointerException: null
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> WARN  [SharedPool-Worker-3] 2016-09-26 07:35:31,247 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-3,5,main]: {}
> java.lang.RuntimeException: java.lang.NullPointerException
>        	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9]
>        	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: java.lang.NullPointerException: null
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.0.9.jar:3.0.9]
>        	... 5 common frames omitted
> This results in queries failing on those tables (not always though, at first). The failures
keep increasing, until total failure after a few days.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message