cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jeff.ji...@crowdstrike.com>
Subject Re: Cassandra Node Restart Stuck in STARTING?
Date Wed, 16 Nov 2016 19:19:59 GMT
What version? 

Is the system doing anything (do you see high CPU / disk usage)?

 

Sometimes restarts will trigger some changes to files on disk that are mostly invisible in
the logs (https://issues.apache.org/jira/browse/CASSANDRA-11163 for example), but it’s usually
during a different part of the startup process (you’d be seeing different log messages),
and would eventually complete. 

 

 

 

From: Daniel Subak <daniel.subak@klaviyo.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, November 16, 2016 at 11:05 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Cassandra Node Restart Stuck in STARTING?

 

Hey everyone,

Ran into an issue running a node restart where "nodetool netstats" reported the node as "STARTING"
with no streams when run locally. "nodetool status" run on other nodes reported that node
as "DN". Both of those were expected. However, tailing the logs, there didn't seem to be anything
noteworthy happening (below are the last few log lines in system.log.) Has anyone seen this
behavior before? We'd love to be able to better monitor what is happening during a restart
if anyone has some information on what happens during this phase! Happy to provide more info
if needed, but even a high level general explanation would provide some clarity

Thanks,

Dan


INFO  [main] 2016-11-16 18:07:55,907 ColumnFamilyStore.java:405 - Initializing system_schema.keyspaces
INFO  [main] 2016-11-16 18:07:55,942 ColumnFamilyStore.java:405 - Initializing system_schema.tables
INFO  [main] 2016-11-16 18:07:55,971 ColumnFamilyStore.java:405 - Initializing system_schema.columns
INFO  [main] 2016-11-16 18:07:55,992 ColumnFamilyStore.java:405 - Initializing system_schema.triggers
INFO  [main] 2016-11-16 18:07:56,010 ColumnFamilyStore.java:405 - Initializing system_schema.dropped_columns
INFO  [main] 2016-11-16 18:07:56,026 ColumnFamilyStore.java:405 - Initializing system_schema.views
INFO  [main] 2016-11-16 18:07:56,047 ColumnFamilyStore.java:405 - Initializing system_schema.types
INFO  [main] 2016-11-16 18:07:56,066 ColumnFamilyStore.java:405 - Initializing system_schema.functions
INFO  [main] 2016-11-16 18:07:56,081 ColumnFamilyStore.java:405 - Initializing system_schema.aggregates
INFO  [main] 2016-11-16 18:07:56,093 ColumnFamilyStore.java:405 - Initializing system_schema.indexes
INFO  [main] 2016-11-16 18:07:56,102 ViewManager.java:139 - Not submitting build tasks for
views in keyspace system_schema as storage service is not initialized


Mime
View raw message