Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4B052200CA7 for ; Wed, 14 Jun 2017 11:55:04 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 49A7E160BEA; Wed, 14 Jun 2017 09:55:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 69F95160BDB for ; Wed, 14 Jun 2017 11:55:03 +0200 (CEST) Received: (qmail 43783 invoked by uid 500); 14 Jun 2017 09:55:02 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 43771 invoked by uid 99); 14 Jun 2017 09:55:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jun 2017 09:55:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 32039CF7B1 for ; Wed, 14 Jun 2017 09:55:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.211 X-Spam-Level: X-Spam-Status: No, score=-99.211 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id GcDXz3Bbr5cR for ; Wed, 14 Jun 2017 09:55:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id EDC055FB8B for ; Wed, 14 Jun 2017 09:55:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5E493E0984 for ; Wed, 14 Jun 2017 09:55:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1521C21D8F for ; Wed, 14 Jun 2017 09:55:00 +0000 (UTC) Date: Wed, 14 Jun 2017 09:55:00 +0000 (UTC) From: "Stefano Ortolani (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 14 Jun 2017 09:55:04 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048886#comment-16048886 ] Stefano Ortolani edited comment on CASSANDRA-13043 at 6/14/17 9:54 AM: ----------------------------------------------------------------------- Update: turned out the commitlog was a false flag. On all other nodes where I found this issue I had no replayed mutations. The missed QUORUMs were client requests being served before the node had full visibility over the cluster. However, there two things that are weird: 1) It's always about a counter mutation (and the distribution of queries is heavily skewed in favour of non-counter queries). 2) The exception happens while the node is updating the topology and before the gossip settles (which is weird since I thought client requests were not allowed until then. was (Author: ostefano): Update: turned out the commitlog was a false flag. On all other nodes where I found this issue I had no replayed mutations. The missed QUORUMs were client requests being served before the node had full visibility over the cluster. > Unable to achieve CL while applying counters from commitlog > ----------------------------------------------------------- > > Key: CASSANDRA-13043 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13043 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Debian > Reporter: Catalin Alexandru Zamfir > > In version 3.9 of Cassandra, we get the following exceptions on the system.log whenever booting an agent. They seem to grow in number with each reboot. Any idea where they come from or what can we do about them? Note that the cluster is healthy (has sufficient live nodes). > {noformat} > 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120 > 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120 > 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread Thread[CounterMutationStage-111,5,main]: {} > 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread Thread[CounterMutationStage-118,5,main]: {} > 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread Thread[CounterMutationStage-164,5,main]: {} > 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread Thread[CounterMutationStage-117,5,main]: {} > 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9] > 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org