cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefano Ortolani (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
Date Mon, 26 Jun 2017 11:24:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062950#comment-16062950
] 

Stefano Ortolani edited comment on CASSANDRA-13043 at 6/26/17 11:23 AM:
------------------------------------------------------------------------

Based on my analysis the request is coming from another node (and not from a client), apparently
the coordinator. I don't understand why the replica node (the node where this exception is
triggered) is checking for Consistency Level though, but I guess it might be a corner case
of how counters are handled. 

[~iamaleksey], I have been told you might have some more insights here re how counters are
handled?

I guess an easy solution would be to delay requests from other nodes.
Would that make sense/being feasible?



was (Author: ostefano):
Based on my analysis the request is coming from another node (and not from a client), apparently
the coordinator. I don't understand why the replica node (the node where this exception is
triggered) is checking for Consistency Level though, but I guess it might be a corner case
of how counters are handled. 

[~iamaleksey], I have been yold you might have some more insights here re how counters are
handled?

I guess an easy solution would be to delay requests from other nodes.
Would that make sense/being feasible?


> Unable to achieve CL while applying counters from commitlog
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-13043
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Debian
>            Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the system.log whenever
booting an agent. They seem to grow in number with each reboot. Any idea where they come from
or what can we do about them? Note that the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-111,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-118,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-164,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-117,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message