cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com>
Subject Re: Nodetool cleanup doesn't work
Date Thu, 11 May 2017 18:19:37 GMT
Thanks Jeff,

I have a C* cluster spread across multiple datacenter.
reason for cleanup : I added multiple nodes to cluster and need to run
cleanup on old nodes so that the redundant data is cleaned-up.

On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa <jjirsa@apache.org> wrote:

>
>
> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
> > Hello,
> >
> > I am running into an issue where *nodetool cleanup *fails to cleanup
> data.
> > We are running 2.1.16 version of Cassandra.
> >
> >
> > [user@host ~]$ nodetool cleanup
> > Aborted cleaning up atleast one column family in keyspace user, check
> > server logs for more information.
> > Aborted cleaning up atleast one column family in keyspace org, check
> server
> > logs for more information.
> > error: nodetool failed, check server logs
> > -- StackTrace --
> > java.lang.RuntimeException: nodetool failed, check server logs
> >         at
> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
> >         at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
> >
> > *Logs:*
> >
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
> > the ring
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
> > the ring
> >
> > All the nodes in the cluster are up and running. We tried doing a rolling
> > restart of all nodes and no luck.
> >
> > After looking at the Cassandra JIRA :
> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
> issue
> > is fixed with 2.2.6 and 3.0 version.
> > While we have plans to upgrade to the latest versions(which might take
> > longer time), does any know if there is any work around to mitigate the
> > issue?
> >
>
> Are you running multiple datacenters, and you just removed a specific
> datacenter from a keyspace (and that's why you want to run cleanup)? If
> that's the case, I fear the fix for 10991 isn't really going to fix it in
> the way you hope (we may need a follow-up jira). What you'll almost
> certainly need to do is remove the data on disk manually, which is quite
> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
> each node in the datacenter for which you removed replication.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>

Mime
View raw message