cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Witschey (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10321) Gossip to dead nodes caused CPU usage to be 100%
Date Wed, 16 Sep 2015 15:29:46 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jim Witschey updated CASSANDRA-10321:
-------------------------------------
    Assignee:     (was: Stefania)

> Gossip to dead nodes caused CPU usage to be 100% 
> -------------------------------------------------
>
>                 Key: CASSANDRA-10321
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10321
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Dikang Gu
>
> For one node, the cpu usage jumped to 100%, and logs are full of:
> 2015-09-14_16:34:45.56407 WARN  16:34:45 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:46.66616 WARN  16:34:46 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:47.76830 WARN  16:34:47 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:48.87043 WARN  16:34:48 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:49.97253 WARN  16:34:49 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:51.07462 WARN  16:34:51 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:52.17669 WARN  16:34:52 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:53.27880 WARN  16:34:53 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:54.38090 WARN  16:34:54 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:55.48301 WARN  16:34:55 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:56.58509 WARN  16:34:56 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:57.68721 WARN  16:34:57 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:58.78932 WARN  16:34:58 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:34:59.89142 WARN  16:34:59 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:00.99352 WARN  16:35:00 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:02.09563 WARN  16:35:02 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:03.19775 WARN  16:35:03 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:04.29982 WARN  16:35:04 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:05.40187 WARN  16:35:05 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:06.50369 WARN  16:35:06 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:07.60577 WARN  16:35:07 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:08.70779 WARN  16:35:08 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:09.80968 WARN  16:35:09 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:10.91157 WARN  16:35:10 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:12.01365 WARN  16:35:12 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:13.11569 WARN  16:35:13 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:14.21757 WARN  16:35:14 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:15.31942 WARN  16:35:15 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:16.42132 WARN  16:35:16 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:17.52332 WARN  16:35:17 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:18.62511 WARN  16:35:18 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:19.72697 WARN  16:35:19 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:20.82872 WARN  16:35:20 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:21.93074 WARN  16:35:21 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:23.03281 WARN  16:35:23 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:24.13478 WARN  16:35:24 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:25.23672 WARN  16:35:25 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:26.33862 WARN  16:35:26 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:27.44068 WARN  16:35:27 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:28.54267 WARN  16:35:28 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:29.64461 WARN  16:35:29 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:30.74646 WARN  16:35:30 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:35:31.84823 WARN  16:35:31 Gossip stage has 32 pending tasks; skipping
status check (no nodes will be marked down)
> 2015-09-14_16:40:09.97371 WARN  16:40:09 /2401:db00:2020:716b:face:0:21:0 has 2311 dropped
hints, because node is down past configured hint window.
> This happened to several nodes in our cluster, but one node each time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message