Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 11 Nov 2015 06:37:11 +0000 (UTC)
From: "Hao Bryan Cheng (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12903277.1444291025000.36852.1447223831074@Atlassian.JIRA>
In-Reply-To: <JIRA.12903277.1444291025000@Atlassian.JIRA>
References: <JIRA.12903277.1444291025000@Atlassian.JIRA>
 <JIRA.12903277.1444291025127@arcas>
Subject: [jira] [Commented] (CASSANDRA-10477) java.lang.AssertionError in
 StorageProxy.submitHint
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000004#comment-15000004 ] 

Hao Bryan Cheng commented on CASSANDRA-10477:
---------------------------------------------

A few additional details:

Unfortunately, I didn't get any data while the issue was happening. Afterwards, netstat, nodetool status, etc. are all nominal.

During the period of time when this node was experiencing difficulty, no other nodes reported any unhealthy hosts. However, we do have our phi convict threshold tuned up from 8 to 10, due to running on AWS.

This event was localized to one node out of 12. Keyspace RF ranges from 3-5. Queries at LOCAL_QUORUM were timing out with insufficient responses.

> java.lang.AssertionError in StorageProxy.submitHint
> ---------------------------------------------------
>
>                 Key: CASSANDRA-10477
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10477
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: CentOS 6, Oracle JVM 1.8.45
>            Reporter: Severin Leonhardt
>            Assignee: Ariel Weisberg
>             Fix For: 2.1.x
>
>
> A few days after updating from 2.0.15 to 2.1.9 we have the following log entry on 2 of 5 machines:
> {noformat}
> ERROR [EXPIRING-MAP-REAPER:1] 2015-10-07 17:01:08,041 CassandraDaemon.java:223 - Exception in thread Thread[EXPIRING-MAP-REAPER:1,5,main]
> java.lang.AssertionError: /192.168.11.88
>         at org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:949) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.net.MessagingService$5.apply(MessagingService.java:383) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.net.MessagingService$5.apply(MessagingService.java:363) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.utils.ExpiringMap$1.run(ExpiringMap.java:98) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {noformat}
> 192.168.11.88 is the broadcast address of the local machine.
> When this is logged the read request latency of the whole cluster becomes very bad, from 6 ms/op to more than 100 ms/op according to OpsCenter. Clients get a lot of timeouts. We need to restart the affected Cassandra node to get back normal read latencies. It seems write latency is not affected.
> Disabling hinted handoff using {{nodetool disablehandoff}} only prevents the assert from being logged. At some point the read latency becomes bad again. Restarting the node where hinted handoff was disabled results in the read latency being better again.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)