cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Lubow (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6604) AssertionError: Verb COUNTER_MUTATION should not legally be dropped
Date Thu, 06 Feb 2014 23:49:19 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13893996#comment-13893996
] 

Eric Lubow commented on CASSANDRA-6604:
---------------------------------------

I'm not entirely sure this is cosmetic.  We have a 20 node cluster running 1.2.13.2 (DSE 3.2.4)
and it is not under a heavy load at all.  We are seeing this simultaneously with tens of thousands
of dropped mutations under higher traffic scenarios and just thousands of mutations dropped
under lower traffic scenarios.  However, it is only happening on a single node.  There is
nothing (visibly) wrong with that node other than higher CPU utilization that the other nodes.
 Even with the higher CPU utilization, it is still not a stressed node and yet we are seeing
the dropped MUTATIONS and this error message (again only on this node).

> AssertionError: Verb COUNTER_MUTATION should not legally be dropped
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-6604
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6604
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Bartłomiej Romański
>             Fix For: 1.2.14, 2.0.5
>
>
> We're seeing the following errors in our logs from time to time (about once an hour):
> {code}
> ERROR [MutationStage:10] 2014-01-19 10:36:26,659 CassandraDaemon.java (line 187) Exception
in thread Thread[MutationStage:10,5,main]
> java.lang.AssertionError: Verb COUNTER_MUTATION should not legally be dropped
>         at org.apache.cassandra.net.MessagingService.incrementDroppedMessages(MessagingService.java:779)
>         at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1925)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> {code}
> They always appear in groups. About 50-100 errors in a row.
> We've got a 12-nodes cluster recently upgraded from 1.2.10 to 2.0.4. It's under pretty
heavy load.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message