cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Nodetool cleanup doesn't work
Date Thu, 11 May 2017 20:11:48 GMT
No, it's not expected, but it's pretty obvious from reading the code
what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-13526





On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:

> Yes I have many keyspaces which are not spread across all the data
> centers(expected by design).
> In this case, is this the expected behavior cleanup will not work for all
> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
> versions?
>
> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
> keyspacename"
>
> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>
>> If you didn't explicitly remove a keyspace from one of your datacenters,
>> the next most likely cause is that you have one keyspace that's NOT
>> replicated to one of the datacenters. You can work around this by running
>> 'nodetool cleanup <ks>' on all of your other keyspaces individually,
>> skipping the one that isn't replicated to that datacenter.
>>
>>
>>
>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>> Thanks Jeff,
>>>
>>> I have a C* cluster spread across multiple datacenter.
>>> reason for cleanup : I added multiple nodes to cluster and need to run
>>> cleanup on old nodes so that the redundant data is cleaned-up.
>>>
>>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa <jjirsa@apache.org> wrote:
>>>
>>>>
>>>>
>>>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
>>>> jaibheemsen@gmail.com> wrote:
>>>> > Hello,
>>>> >
>>>> > I am running into an issue where *nodetool cleanup *fails to cleanup
>>>> data.
>>>> > We are running 2.1.16 version of Cassandra.
>>>> >
>>>> >
>>>> > [user@host ~]$ nodetool cleanup
>>>> > Aborted cleaning up atleast one column family in keyspace user, check
>>>> > server logs for more information.
>>>> > Aborted cleaning up atleast one column family in keyspace org, check
>>>> server
>>>> > logs for more information.
>>>> > error: nodetool failed, check server logs
>>>> > -- StackTrace --
>>>> > java.lang.RuntimeException: nodetool failed, check server logs
>>>> >         at
>>>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool
>>>> .java:294)
>>>> >         at org.apache.cassandra.tools.Nod
>>>> eTool.main(NodeTool.java:206)
>>>> >
>>>> > *Logs:*
>>>> >
>>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
>>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>>> joined
>>>> > the ring
>>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
>>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>>> joined
>>>> > the ring
>>>> >
>>>> > All the nodes in the cluster are up and running. We tried doing a
>>>> rolling
>>>> > restart of all nodes and no luck.
>>>> >
>>>> > After looking at the Cassandra JIRA :
>>>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
>>>> issue
>>>> > is fixed with 2.2.6 and 3.0 version.
>>>> > While we have plans to upgrade to the latest versions(which might take
>>>> > longer time), does any know if there is any work around to mitigate
>>>> the
>>>> > issue?
>>>> >
>>>>
>>>> Are you running multiple datacenters, and you just removed a specific
>>>> datacenter from a keyspace (and that's why you want to run cleanup)? If
>>>> that's the case, I fear the fix for 10991 isn't really going to fix it in
>>>> the way you hope (we may need a follow-up jira). What you'll almost
>>>> certainly need to do is remove the data on disk manually, which is quite
>>>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
>>>> each node in the datacenter for which you removed replication.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>
>>>>
>>>
>>
>

Mime
View raw message