cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com>
Subject Re: Nodetool cleanup doesn't work
Date Thu, 11 May 2017 20:33:30 GMT
ok thank you

On Thu, May 11, 2017 at 1:11 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:

> No, it's not expected, but it's pretty obvious from reading the code
> what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-
> 13526
>
>
>
>
>
> On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> Yes I have many keyspaces which are not spread across all the data
>> centers(expected by design).
>> In this case, is this the expected behavior cleanup will not work for all
>> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
>> versions?
>>
>> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
>> keyspacename"
>>
>> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>>
>>> If you didn't explicitly remove a keyspace from one of your datacenters,
>>> the next most likely cause is that you have one keyspace that's NOT
>>> replicated to one of the datacenters. You can work around this by running
>>> 'nodetool cleanup <ks>' on all of your other keyspaces individually,
>>> skipping the one that isn't replicated to that datacenter.
>>>
>>>
>>>
>>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
>>> jaibheemsen@gmail.com> wrote:
>>>
>>>> Thanks Jeff,
>>>>
>>>> I have a C* cluster spread across multiple datacenter.
>>>> reason for cleanup : I added multiple nodes to cluster and need to run
>>>> cleanup on old nodes so that the redundant data is cleaned-up.
>>>>
>>>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa <jjirsa@apache.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
>>>>> jaibheemsen@gmail.com> wrote:
>>>>> > Hello,
>>>>> >
>>>>> > I am running into an issue where *nodetool cleanup *fails to cleanup
>>>>> data.
>>>>> > We are running 2.1.16 version of Cassandra.
>>>>> >
>>>>> >
>>>>> > [user@host ~]$ nodetool cleanup
>>>>> > Aborted cleaning up atleast one column family in keyspace user,
check
>>>>> > server logs for more information.
>>>>> > Aborted cleaning up atleast one column family in keyspace org, check
>>>>> server
>>>>> > logs for more information.
>>>>> > error: nodetool failed, check server logs
>>>>> > -- StackTrace --
>>>>> > java.lang.RuntimeException: nodetool failed, check server logs
>>>>> >         at
>>>>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool
>>>>> .java:294)
>>>>> >         at org.apache.cassandra.tools.Nod
>>>>> eTool.main(NodeTool.java:206)
>>>>> >
>>>>> > *Logs:*
>>>>> >
>>>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
>>>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>>>> joined
>>>>> > the ring
>>>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
>>>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>>>> joined
>>>>> > the ring
>>>>> >
>>>>> > All the nodes in the cluster are up and running. We tried doing
a
>>>>> rolling
>>>>> > restart of all nodes and no luck.
>>>>> >
>>>>> > After looking at the Cassandra JIRA :
>>>>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like
>>>>> the issue
>>>>> > is fixed with 2.2.6 and 3.0 version.
>>>>> > While we have plans to upgrade to the latest versions(which might
>>>>> take
>>>>> > longer time), does any know if there is any work around to mitigate
>>>>> the
>>>>> > issue?
>>>>> >
>>>>>
>>>>> Are you running multiple datacenters, and you just removed a specific
>>>>> datacenter from a keyspace (and that's why you want to run cleanup)?
If
>>>>> that's the case, I fear the fix for 10991 isn't really going to fix it
in
>>>>> the way you hope (we may need a follow-up jira). What you'll almost
>>>>> certainly need to do is remove the data on disk manually, which is quite
>>>>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
>>>>> each node in the datacenter for which you removed replication.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message