cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13043) UnavailabeException caused by counter writes forwarded to leaders without complete cluster view
Date Thu, 21 Sep 2017 12:54:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174690#comment-16174690
] 

Sylvain Lebresne commented on CASSANDRA-13043:
----------------------------------------------

To me, it sounds like the problem we have is that we move endpoints from 'pending' to 'natural'
endpoints too quickly after bootstrap, before the node is actually fully ready, and that this
is what we should be fixing (basically, {{getLiveNaturalEndpoints}} shouldn't return node
that are note truly ready, and not knowing the ring yet is not ready to me). Maybe counters
are the only case where this is a problem today, but hand-fixing it only there feels like
we'll run into other problems like that in the future. That being said, the whole boostrap/gossip-settling
code is not the part I', the most confortable with so I don't know how easy fixing that is
(or if it's even possible/reasonable to do in 3.x).

Other than that, I agree with you concern. If we want to stick with the current approach at
least as far 3.x goes for simplicity, then maybe we can simply advertise rpc-ready at the
end of {{CassandraDaemon.start}} but in all case? In fact, I'm not sure why we only set it
for the native protocol, and that might simply be an oversight. At the very least, thrift
is literally referred to "rpc" in the yaml, so not setting {{RPC_READY}} for it feels wrong.
And going a step further, if you have explicitly asked to not start any of the client server,
then it wouldn't be a lie to set {{RPC_READY}} at the end of {{start()}} in the sense of "every
RPC servers you asked us to start is ready". Anyway, maybe someone rely on this being only
set for the native protocol, I genuinely have no clue.

> UnavailabeException caused by counter writes forwarded to leaders without complete cluster
view
> -----------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-13043
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Coordination
>         Environment: Debian
>            Reporter: Catalin Alexandru Zamfir
>            Assignee: Stefano Ortolani
>            Priority: Minor
>             Fix For: 3.0.15, 3.11.1
>
>         Attachments: 13043-3.0.patch, patch.diff
>
>
> In version 3.9 of Cassandra, we get the following exceptions on the system.log whenever
booting an agent. They seem to grow in number with each reboot. Any idea where they come from
or what can we do about them? Note that the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-111,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-118,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-164,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-117,5,main]:
{}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message