cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinath Reddy <ksre...@gmail.com>
Subject Re: Error when running nodetool cleanup after adding a new node to a cluster
Date Thu, 09 Feb 2017 08:53:59 GMT
Alex,

Thanks for reply.  I will try the workaround and post an update.

Regards,

Srinath Reddy

> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <oleksandr.shulgin@zalando.de> wrote:
> 
> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ksreddy@gmail.com <mailto:ksreddy@gmail.com>>
wrote:
> Hi,
> 
> Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this
error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster.
> 
> Cassandra version is 2.2.8
> 
> nodetool cleanup
> error: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server.  See log for stacktrace.
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server.  See log for stacktrace.
> 	at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
> 	at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
> 	at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
> 	at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
> 	at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
> 	at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
> 	at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
> 
> Hi,
> 
> From the above stacktrace it looks like you're hitting the following TODO item:
> 
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
<https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282>
> 
> That is, nodetool needs to know concurrent_compactors setting's value before starting
cleanup, but doesn't use JMX and tries to parse the configuration file instead.  That fails
because your custom SeedProvider class is not on classpath for nodetool.
> 
> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be found by java
when running nodetool script, see https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
<https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108>
> 
> Proper fix: get rid of the TODO and really query the value using JMX, especially since
the latest tick-tock release of Cassandra (3.10) added a way to modify it with JMX.
> 
> --
> Alex


Mime
View raw message