cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Subak <daniel.su...@klaviyo.com>
Subject Re: Cassandra Node Restart Stuck in STARTING?
Date Wed, 16 Nov 2016 19:27:00 GMT
We're on Cassandra 3.7, running on Ubuntu 14.04.
In terms of system utilization, we saw one Cassandra process which was
using 100% CPU, but overall load was very low on the box. Disk utilization
was largely nominal.

On Wed, Nov 16, 2016 at 2:19 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com>
wrote:

> What version?
>
> Is the system doing anything (do you see high CPU / disk usage)?
>
>
>
> Sometimes restarts will trigger some changes to files on disk that are
> mostly invisible in the logs (https://issues.apache.org/
> jira/browse/CASSANDRA-11163 for example), but it’s usually during a
> different part of the startup process (you’d be seeing different log
> messages), and would eventually complete.
>
>
>
>
>
>
>
> *From: *Daniel Subak <daniel.subak@klaviyo.com>
> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Date: *Wednesday, November 16, 2016 at 11:05 AM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *Cassandra Node Restart Stuck in STARTING?
>
>
>
> Hey everyone,
>
> Ran into an issue running a node restart where "nodetool netstats"
> reported the node as "STARTING" with no streams when run locally. "nodetool
> status" run on other nodes reported that node as "DN". Both of those were
> expected. However, tailing the logs, there didn't seem to be anything
> noteworthy happening (below are the last few log lines in system.log.) Has
> anyone seen this behavior before? We'd love to be able to better monitor
> what is happening during a restart if anyone has some information on what
> happens during this phase! Happy to provide more info if needed, but even a
> high level general explanation would provide some clarity
>
> Thanks,
>
> Dan
>
>
> INFO  [main] 2016-11-16 18:07:55,907 ColumnFamilyStore.java:405 -
> Initializing system_schema.keyspaces
> INFO  [main] 2016-11-16 18:07:55,942 ColumnFamilyStore.java:405 -
> Initializing system_schema.tables
> INFO  [main] 2016-11-16 18:07:55,971 ColumnFamilyStore.java:405 -
> Initializing system_schema.columns
> INFO  [main] 2016-11-16 18:07:55,992 ColumnFamilyStore.java:405 -
> Initializing system_schema.triggers
> INFO  [main] 2016-11-16 18:07:56,010 ColumnFamilyStore.java:405 -
> Initializing system_schema.dropped_columns
> INFO  [main] 2016-11-16 18:07:56,026 ColumnFamilyStore.java:405 -
> Initializing system_schema.views
> INFO  [main] 2016-11-16 18:07:56,047 ColumnFamilyStore.java:405 -
> Initializing system_schema.types
> INFO  [main] 2016-11-16 18:07:56,066 ColumnFamilyStore.java:405 -
> Initializing system_schema.functions
> INFO  [main] 2016-11-16 18:07:56,081 ColumnFamilyStore.java:405 -
> Initializing system_schema.aggregates
> INFO  [main] 2016-11-16 18:07:56,093 ColumnFamilyStore.java:405 -
> Initializing system_schema.indexes
> INFO  [main] 2016-11-16 18:07:56,102 ViewManager.java:139 - Not submitting
> build tasks for views in keyspace system_schema as storage service is not
> initialized
>

Mime
View raw message