cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10321) Gossip to dead nodes caused CPU usage to be 100%
Date Wed, 16 Sep 2015 15:36:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790565#comment-14790565
] 

Brandon Williams commented on CASSANDRA-10321:
----------------------------------------------

Without knowing what threads were using the cpu we have nowhere to go.  It looks like gossip
was doing its job and skipping the status check to avoid compounding an existing problem.

> Gossip to dead nodes caused CPU usage to be 100% 
> -------------------------------------------------
>
>                 Key: CASSANDRA-10321
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10321
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Dikang Gu
>
> For one node, the cpu usage jumped to 100%, and logs are full of:
> 2015-09-14_16:34:45.56407 WARN  16:34:45 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:46.66616 WARN  16:34:46 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:47.76830 WARN  16:34:47 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:48.87043 WARN  16:34:48 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:49.97253 WARN  16:34:49 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:51.07462 WARN  16:34:51 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:52.17669 WARN  16:34:52 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:53.27880 WARN  16:34:53 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:54.38090 WARN  16:34:54 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:55.48301 WARN  16:34:55 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:56.58509 WARN  16:34:56 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:57.68721 WARN  16:34:57 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:58.78932 WARN  16:34:58 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:59.89142 WARN  16:34:59 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:00.99352 WARN  16:35:00 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:02.09563 WARN  16:35:02 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:03.19775 WARN  16:35:03 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:04.29982 WARN  16:35:04 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:05.40187 WARN  16:35:05 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:06.50369 WARN  16:35:06 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:07.60577 WARN  16:35:07 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:08.70779 WARN  16:35:08 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:09.80968 WARN  16:35:09 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:10.91157 WARN  16:35:10 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:12.01365 WARN  16:35:12 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:13.11569 WARN  16:35:13 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:14.21757 WARN  16:35:14 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:15.31942 WARN  16:35:15 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:16.42132 WARN  16:35:16 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:17.52332 WARN  16:35:17 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:18.62511 WARN  16:35:18 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:19.72697 WARN  16:35:19 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:20.82872 WARN  16:35:20 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:21.93074 WARN  16:35:21 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:23.03281 WARN  16:35:23 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:24.13478 WARN  16:35:24 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:25.23672 WARN  16:35:25 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:26.33862 WARN  16:35:26 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:27.44068 WARN  16:35:27 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:28.54267 WARN  16:35:28 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:29.64461 WARN  16:35:29 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:30.74646 WARN  16:35:30 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:31.84823 WARN  16:35:31 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:40:09.97371 WARN  16:40:09 /2401:db00:2020:716b:face:0:21:0 has 2311 dropped
hints, because node is down past configured hint window.
> This happened to several nodes in our cluster, but one node each time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message