cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rob Emery (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10041) "timeout during write query at consistency ONE" when updating counter at consistency QUORUM and 2 of 3 nodes alive
Date Thu, 07 Apr 2016 09:51:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230017#comment-15230017
] 

Rob Emery commented on CASSANDRA-10041:
---------------------------------------

Interesting; from our perspective the nodes that intermittently throw this exception are not
actually down. They are delivering other workloads without any apparent problem. We just get
a few of these per day and have been unable to track down any cause (there does not appear
to be any IO pressure or GC pauses etc); therefore for our situation all information, no matter
how small, is vital for us to be able to diagnose the problem. Currently we have no way of
tracking the point during the mutation that the issue is actually occurring. 

> "timeout during write query at consistency ONE" when updating counter at consistency
QUORUM and 2 of 3 nodes alive
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10041
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: centos 6.6 server, java version "1.8.0_45", cassandra 2.1.8, 3 machines,
keyspace with replication factor 3
>            Reporter: Anton Lebedevich
>             Fix For: 2.1.x
>
>
> Test scenario is: kill -9 one node, wait 60 seconds, start it back, wait till it becomes
available, wait 120 seconds (during that time all 3 nodes are up), repeat with the next node.
Application reads from one table and updates counters in another table with consistency QUORUM.
When one node out of 3 is killed application logs this exception for several seconds:
> {noformat}
> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout
during write query at consistency ONE (1 replica were required but only 0 acknowledged the
write)
>         at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:57) ~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
>         at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37) ~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
>         at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:204)
~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
>         at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:195)
~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
>         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
[io.netty.netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         ... 13 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message