Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 32DF2200C63 for ; Thu, 11 May 2017 22:12:19 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 316DE160BC7; Thu, 11 May 2017 20:12:19 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2A2D1160BB3 for ; Thu, 11 May 2017 22:12:18 +0200 (CEST) Received: (qmail 8308 invoked by uid 500); 11 May 2017 20:12:16 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 8298 invoked by uid 99); 11 May 2017 20:12:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 May 2017 20:12:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EA0C0C03A8 for ; Thu, 11 May 2017 20:12:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 9hJ8PQQ0ciXI for ; Thu, 11 May 2017 20:12:10 +0000 (UTC) Received: from mail-qt0-f179.google.com (mail-qt0-f179.google.com [209.85.216.179]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 13C965FAE1 for ; Thu, 11 May 2017 20:12:10 +0000 (UTC) Received: by mail-qt0-f179.google.com with SMTP id m91so26689524qte.3 for ; Thu, 11 May 2017 13:12:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=v+XkCcQnVnIwpwVUKQrfBPuEaQZRZTmi0GuOMaAPixI=; b=OvVwnkEXr50K+81+xPbhjyRVjYePj2HXvoHgR3nNWwQgjK0tHlRzTbJ/bQe/jS9ZWj G/AyP2CYvs8+4zACdGm/dsSKcxVr3GXG0mzMzfV6ptDFqEyl9Id2jcIOrXz1i8cLOjsA FUia/dy6KNaNj37OUJEP+VIB8I+hEePKQcwaXp8qHsg4rixYSnAUef9xwqlcoK5WFen0 +JOWab5bkeoF//2XpjI56yc2Y7aGaQ8hWPcvHhoiRsPD3xr6ARKdUog/HA2Kbig38pqf QLFrxWToTfIn9oqIBO2iYWe6YuBqo4GMQv9ZY4Vr9kh+gHg87PRtFJTgZbu7We16Z0VJ x7Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=v+XkCcQnVnIwpwVUKQrfBPuEaQZRZTmi0GuOMaAPixI=; b=EmfWzXXHi4sPQ/UC+iEtBOpmY5ORpkgsAdDupBN7wmR2FVwnQEN80SbQsbyczNY08s rrNq/nmiPFec3jDeVCMUQOYUs5gZtot+7E1EJc5B8nFXNapPrE4lAWttu4xQKQP9My6J NhXYxvFr6rg/yTFuuYTroUDpfjKehQokXH3W9emxGmryA8olh9IzgTvEV6K47nC+fFRq WRENpDk9xUXfuZH5wrWBRgH/NHTS/XBrIW6LvjWnWzWjvw9bOUMQcYOcCorzinEvLWCT DCbJBXSabNi/dOWYhiKv4KVBRHCz3vpIbYRO+yFO+FkUHnzyaYN3nYIk0WEO/ugORPeG 9KGw== X-Gm-Message-State: AODbwcA/Y6XOVlLYGQFj9JXZGmQ1tocsQi6NCMC4esJjOqUP0cawxUqW cT23ouLY9zJWPSqGNPKDVTLjzPECwQ== X-Received: by 10.200.38.76 with SMTP id v12mr362436qtv.257.1494533529083; Thu, 11 May 2017 13:12:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.51.153 with HTTP; Thu, 11 May 2017 13:11:48 -0700 (PDT) In-Reply-To: References: From: Jeff Jirsa Date: Thu, 11 May 2017 13:11:48 -0700 Message-ID: Subject: Re: Nodetool cleanup doesn't work To: Jai Bheemsen Rao Dhanwada Cc: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="001a11404c781b3f5b054f4534ca" archived-at: Thu, 11 May 2017 20:12:19 -0000 --001a11404c781b3f5b054f4534ca Content-Type: text/plain; charset="UTF-8" No, it's not expected, but it's pretty obvious from reading the code what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-13526 On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada < jaibheemsen@gmail.com> wrote: > Yes I have many keyspaces which are not spread across all the data > centers(expected by design). > In this case, is this the expected behavior cleanup will not work for all > the keyspaces(nodetool cleanup)? is it going to be fixed in the latest > versions? > > P.S: Thanks for the tip, I can workaround this by "nodetool cleanup > keyspacename" > > On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa wrote: > >> If you didn't explicitly remove a keyspace from one of your datacenters, >> the next most likely cause is that you have one keyspace that's NOT >> replicated to one of the datacenters. You can work around this by running >> 'nodetool cleanup ' on all of your other keyspaces individually, >> skipping the one that isn't replicated to that datacenter. >> >> >> >> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada < >> jaibheemsen@gmail.com> wrote: >> >>> Thanks Jeff, >>> >>> I have a C* cluster spread across multiple datacenter. >>> reason for cleanup : I added multiple nodes to cluster and need to run >>> cleanup on old nodes so that the redundant data is cleaned-up. >>> >>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa wrote: >>> >>>> >>>> >>>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada < >>>> jaibheemsen@gmail.com> wrote: >>>> > Hello, >>>> > >>>> > I am running into an issue where *nodetool cleanup *fails to cleanup >>>> data. >>>> > We are running 2.1.16 version of Cassandra. >>>> > >>>> > >>>> > [user@host ~]$ nodetool cleanup >>>> > Aborted cleaning up atleast one column family in keyspace user, check >>>> > server logs for more information. >>>> > Aborted cleaning up atleast one column family in keyspace org, check >>>> server >>>> > logs for more information. >>>> > error: nodetool failed, check server logs >>>> > -- StackTrace -- >>>> > java.lang.RuntimeException: nodetool failed, check server logs >>>> > at >>>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool >>>> .java:294) >>>> > at org.apache.cassandra.tools.Nod >>>> eTool.main(NodeTool.java:206) >>>> > >>>> > *Logs:* >>>> > >>>> > INFO [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987 >>>> > CompactionManager.java:415 - Cleanup cannot run before a node has >>>> joined >>>> > the ring >>>> > INFO [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010 >>>> > CompactionManager.java:415 - Cleanup cannot run before a node has >>>> joined >>>> > the ring >>>> > >>>> > All the nodes in the cluster are up and running. We tried doing a >>>> rolling >>>> > restart of all nodes and no luck. >>>> > >>>> > After looking at the Cassandra JIRA : >>>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the >>>> issue >>>> > is fixed with 2.2.6 and 3.0 version. >>>> > While we have plans to upgrade to the latest versions(which might take >>>> > longer time), does any know if there is any work around to mitigate >>>> the >>>> > issue? >>>> > >>>> >>>> Are you running multiple datacenters, and you just removed a specific >>>> datacenter from a keyspace (and that's why you want to run cleanup)? If >>>> that's the case, I fear the fix for 10991 isn't really going to fix it in >>>> the way you hope (we may need a follow-up jira). What you'll almost >>>> certainly need to do is remove the data on disk manually, which is quite >>>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start >>>> each node in the datacenter for which you removed replication. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org >>>> For additional commands, e-mail: user-help@cassandra.apache.org >>>> >>>> >>> >> > --001a11404c781b3f5b054f4534ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
No, it's not expected, but it's pretty obvious fro= m reading the code what'll happen. Opened=C2=A0https://issues.apache.org/jira/br= owse/CASSANDRA-13526





On= Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <jaibheemsen= @gmail.com> wrote:
Yes I have many keyspaces which are not spread across all the data= centers(expected by design).
In this case, is this the expected behavi= or cleanup will not work for all the keyspaces(nodetool cleanup)? is it goi= ng to be fixed in the latest versions?

P.S: Thanks= for the tip, I can workaround this by "nodetool cleanup keyspacename&= quot;

On Thu, May 11, 2017 at 12:11 PM, J= eff Jirsa <jjirsa@gmail.com> wrote:
If you didn't explicitly remove a keyspace f= rom one of your datacenters, the next most likely cause is that you have on= e keyspace that's NOT replicated to one of the datacenters. You can wor= k around this by running 'nodetool cleanup <ks>' on all of yo= ur other keyspaces individually, skipping the one that isn't replicated= to that datacenter.



On Thu, May 11, 2017 at 11:= 19 AM, Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com>= wrote:
Thanks Jeff,
I have a C* cluster spread across multiple datacenter.
reason for cleanup : I added multiple nodes to cluster and need to ru= n cleanup on old nodes so that the redundant data is cleaned-up.

On Thu, May 11, = 2017 at 11:08 AM, Jeff Jirsa <jjirsa@apache.org> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">

On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com> wr= ote:
> Hello,
>
> I am running into an issue where *nodetool cleanup *fails to cl= eanup data.
> We are running 2.1.16 version of Cassandra.
>
>
> [user@host ~]$ nodetool cleanup
> Aborted cleaning up atleast one column family in keyspace user, check<= br> > server logs for more information.
> Aborted cleaning up atleast one column family in keyspace org, check s= erver
> logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool= .java:294)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
>
> *Logs:*
>
> INFO=C2=A0 [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987 > CompactionManager.java:415 - Cleanup cannot run before a node has join= ed
> the ring
> INFO=C2=A0 [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010 > CompactionManager.java:415 - Cleanup cannot run before a node has join= ed
> the ring
>
> All the nodes in the cluster are up and running. We tried doing a roll= ing
> restart of all nodes and no luck.
>
> After looking at the Cassandra JIRA :
> https://issues.apache.org/jira/brows= e/CASSANDRA-10991 looks like the issue
> is fixed with 2.2.6 and 3.0 version.
> While we have plans to upgrade to the latest versions(which might take=
> longer time), does any know if there is any work around to mitigate th= e
> issue?
>

Are you running multiple datacenters, and you just removed a specifi= c datacenter from a keyspace (and that's why you want to run cleanup)? = If that's the case, I fear the fix for 10991 isn't really going to = fix it in the way you hope (we may need a follow-up jira). What you'll = almost certainly need to do is remove the data on disk manually, which is q= uite unfortunate as it'll require you to stop+delete-data-for-that-keys= pace+start each node in the datacenter for which you removed replicati= on.

-----------------------------------------------------------------= ----
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org





--001a11404c781b3f5b054f4534ca--