Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D7D0B10A6B for ; Fri, 28 Mar 2014 14:31:58 +0000 (UTC) Received: (qmail 45047 invoked by uid 500); 28 Mar 2014 14:31:58 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 44397 invoked by uid 500); 28 Mar 2014 14:31:50 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 44197 invoked by uid 99); 28 Mar 2014 14:31:39 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Mar 2014 14:31:39 +0000 Date: Fri, 28 Mar 2014 14:31:37 +0000 (UTC) From: "Keith Wright (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: ------------------------------------ Attachment: cassandra.yaml Cassandra.yaml attached. We restarted the nodes where this was occurring so I don't have a good JStack at this time however I will attach it when it re-occurs. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > ---------------------------------------------------------------------- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Keith Wright > Attachments: cassandra.yaml > > > We are using the datastax driver and have seen the following exceptions appear periodically in our cluster in "groups". The client will gradually start sending less write requests to the nodes where this occurs the most often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! (CallbackInfo(target=/10.106.160.84, callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 CassandraDaemon.java (line 196) Exception in thread Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! (CallbackInfo(target=/10.106.160.84, callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)