cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Wright (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
Date Fri, 28 Mar 2014 15:30:17 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Keith Wright updated CASSANDRA-6948:
------------------------------------

    Attachment: Screen Shot 2014-03-28 at 11.29.24 AM.png
                Screen Shot 2014-03-28 at 11.27.56 AM.png

Its been up and running for ~2 weeks.  It started showing the issue again ~3 hours ago.  I
have attached a graph showing read requests for the last week as compared to another server.
 You can see that they were lower for this server until ~3 days ago when they recovered on
their own (we did not restart the node but we did restart clients that day which likely caused
the recovery).

I have also attached read/write request volume.  Note we're using RF 3 for all keyspaces.

> Native & ReplicateOnWriteStage AssertionError: Callback already exists
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-6948
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6948
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Keith Wright
>            Assignee: Benedict
>         Attachments: Screen Shot 2014-03-28 at 11.27.56 AM.png, Screen Shot 2014-03-28
at 11.29.24 AM.png, cassandra.yaml, logs.tar.gz, system.log.1.gz, system.log.gz
>
>
> We are using the datastax driver and have seen the following exceptions appear periodically
in our cluster in "groups".  The client will gradually start sending less write requests to
the nodes where this occurs the most often.  This may be related to CASSANDRA-6476.  Exception
from logs:
> ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 ErrorMessage.java (line
222) Unexpected exception during request
> java.lang.AssertionError: Callback already exists for id -1665979622! (CallbackInfo(target=/10.106.160.84,
callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b))
> 	at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549)
> 	at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601)
> 	at org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984)
> 	at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449)
> 	at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524)
> 	at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521)
> 	at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505)
> 	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
> 	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358)
> 	at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131)
> 	at org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
> 	at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> 	at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 CassandraDaemon.java (line
196) Exception in thread Thread[ReplicateOnWriteStage:102766,5,main]
> java.lang.AssertionError: Callback already exists for id -1665979620! (CallbackInfo(target=/10.106.160.84,
callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b))
> 	at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549)
> 	at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601)
> 	at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806)
> 	at org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074)
> 	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message