Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 18 Apr 2014 10:12:16 +0000 (UTC)
From: "sankalp kohli (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12684100.1386793255581.138069.1397815936800@arcas>
In-Reply-To: <JIRA.12684100.1386793255581@arcas>
References: <JIRA.12684100.1386793255581@arcas>
Subject: [jira] [Commented] (CASSANDRA-6476) Assertion error in
 MessagingService.addCallback
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973941#comment-13973941 ] 

sankalp kohli commented on CASSANDRA-6476:
------------------------------------------

We saw these asserts on one node for sometime and then it went away. This is 1.2.15. We had some network problem around the same time. Don't know whether that is related. 
java.lang.AssertionError
	at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:541)
	at org.apache.cassandra.service.StorageProxy.sendMessagesToOneDCInternal(StorageProxy.java:638)
	at org.apache.cassandra.service.StorageProxy.sendMessagesToOneDC(StorageProxy.java:603)
	at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:530)
	at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:121)
	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:384)
	at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:191)
	at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:866)
	at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:849)
	at org.apache.cassandra.thrift.CassandraServer.internal_remove(CassandraServer.java:813)
	at org.apache.cassandra.thrift.CassandraServer.remove(CassandraServer.java:834)
	at org.apache.cassandra.thrift.Cassandra$Processor$remove.getResult(Cassandra.java:3642)
	at org.apache.cassandra.thrift.Cassandra$Processor$remove.getResult(Cassandra.java:3630)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

> Assertion error in MessagingService.addCallback
> -----------------------------------------------
>
>                 Key: CASSANDRA-6476
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6476
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 2.0.2 DCE
>            Reporter: Theo Hultberg
>            Assignee: Sylvain Lebresne
>
> Two of the three Cassandra nodes in one of our clusters just started behaving very strange about an hour ago. Within a minute of each other they started logging AssertionErrors (see stack traces here: https://gist.github.com/iconara/7917438) over and over again. The client lost connection with the nodes at roughly the same time. The nodes were still up, and even if no clients were connected to them they continued logging the same errors over and over.
> The errors are in the native transport (specifically MessagingService.addCallback) which makes me suspect that it has something to do with a test that we started running this afternoon. I've just implemented support for frame compression in my CQL driver cql-rb. About two hours before this happened I deployed a version of the application which enabled Snappy compression on all frames larger than 64 bytes. It's not impossible that there is a bug somewhere in the driver or compression library that caused this -- but at the same time, it feels like it shouldn't be possible to make C* a zombie with a bad frame.
> Restarting seems to have got them back running again, but I suspect they will go down again sooner or later.


--
This message was sent by Atlassian JIRA
(v6.2#6252)