Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id F304B200B9F for ; Mon, 26 Sep 2016 11:15:22 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id F1E59160AC8; Mon, 26 Sep 2016 09:15:22 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E112C160AE3 for ; Mon, 26 Sep 2016 11:15:21 +0200 (CEST) Received: (qmail 74899 invoked by uid 500); 26 Sep 2016 09:15:21 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 74716 invoked by uid 99); 26 Sep 2016 09:15:20 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Sep 2016 09:15:20 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id CF8562C044E for ; Mon, 26 Sep 2016 09:15:20 +0000 (UTC) Date: Mon, 26 Sep 2016 09:15:20 +0000 (UTC) From: "Stefania (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-12706) Exception supposedly after ttl expires and compaction occurs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 26 Sep 2016 09:15:23 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522538#comment-15522538 ] Stefania commented on CASSANDRA-12706: -------------------------------------- I think this very simple [patch|https://github.com/stef1927/cassandra/commits/12706-3.0] should fix it but, could you share your schema and the exact CQL query syntax so that we can try to reproduce the problem? This should be reproducible with the same slice type, which depends on the CQL, and when the tombstones exceed the warning threshold - because of the code in {{ReadCommand.MetricsRecorder.onClose}}: {code} boolean warnTombstones = tombstones > warningThreshold && respectTombstoneThresholds; if (warnTombstones) { String msg = String.format("Read %d live rows and %d tombstone cells for query %1.512s (see tombstone_warn_threshold)", liveRows, tombstones, ReadCommand.this.toCQLString()); ClientWarn.instance.warn(msg); logger.warn(msg); } {code} The NPE is ultimately caused by the call to {{toCQLString()}}. {{respectTombstoneThresholds}} is always true for non system tables, whilst {{warningThreshold}} comes from the tombstone warning threshold in cassandra.yaml. > Exception supposedly after ttl expires and compaction occurs > ------------------------------------------------------------ > > Key: CASSANDRA-12706 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12706 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Linux amd64 Amazon default AMI, > Java 1.8.0_122 oracle jdk > Reporter: Nikhil Sharma > Priority: Critical > > After TTL (1 week) expires and compaction occurs. We start seeing: > WARN [SharedPool-Worker-1] 2016-09-26 07:35:17,420 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {} > java.lang.NullPointerException: null > at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-3.0.9.jar:3.0.9] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] > WARN [SharedPool-Worker-3] 2016-09-26 07:35:31,247 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-3,5,main]: {} > java.lang.RuntimeException: java.lang.NullPointerException > at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470) ~[apache-cassandra-3.0.9.jar:3.0.9] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.9.jar:3.0.9] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] > Caused by: java.lang.NullPointerException: null > at org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797) ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) ~[apache-cassandra-3.0.9.jar:3.0.9] > ... 5 common frames omitted > This results in queries failing on those tables (not always though, at first). The failures keep increasing, until total failure after a few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)