Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E1483FB29 for ; Thu, 21 Mar 2013 22:22:23 +0000 (UTC) Received: (qmail 15808 invoked by uid 500); 21 Mar 2013 22:22:21 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15779 invoked by uid 500); 21 Mar 2013 22:22:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15771 invoked by uid 99); 21 Mar 2013 22:22:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Mar 2013 22:22:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bench@instructure.com designates 209.85.160.43 as permitted sender) Received: from [209.85.160.43] (HELO mail-pb0-f43.google.com) (209.85.160.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Mar 2013 22:22:14 +0000 Received: by mail-pb0-f43.google.com with SMTP id md12so2588391pbc.30 for ; Thu, 21 Mar 2013 15:21:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:mime-version:content-type:subject:date:in-reply-to :to:references:message-id:x-mailer:x-gm-message-state; bh=7LXst/zr6l6pyeHiivBa2GG9oYH2NAESZRqJN4m2apg=; b=nsED/KGAIqtE21UVsebSt9Tew9VQXnG3QpfIUgB4iV9nl3EtCenu4881Fsk8qQ4gOF XWjbbFHmzjkkfhYAhA+0OZfxG0/A49ufDZL1RNE+m9UysYlwKPCbmCOcYw+TkP6z0wGZ NpuJhw+d+Gyi4GMcA8DqVeI/hbHLCZbeNTSPi0mUG+xg9TmwVABo0jmnCUocE2cTN8tJ /hn9QOQxMjo9V3c7Tx2tU/5aTm1kPptNhOo3mBw7xQOEI5ronzgjvQpro6USHFFlVHIq 6Odh7CNRxtzlr4DzUWjzdrmmHPdABp87eLDqfBz8c2DCMII3zvSDBDQpHPp/hQJJigdB Bpow== X-Received: by 10.66.188.99 with SMTP id fz3mr17653360pac.0.1363904513471; Thu, 21 Mar 2013 15:21:53 -0700 (PDT) Received: from [10.2.3.2] (red.silentmedia.com. [70.90.189.53]) by mx.google.com with ESMTPS id 4sm7481811pbn.23.2013.03.21.15.21.50 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 21 Mar 2013 15:21:51 -0700 (PDT) From: Ben Chobot Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: multipart/alternative; boundary=Apple-Mail-31--949747287 Subject: Re: removing old nodes Date: Thu, 21 Mar 2013 15:21:49 -0700 In-Reply-To: To: user@cassandra.apache.org References: <85199489-C408-462C-ACBF-A9F25997FFDD@instructure.com> Message-Id: <83E4FB25-FBB5-4CE4-A8C7-038267A64009@instructure.com> X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQm2DbHUF5etKvXnOZAzkPUyI6/mRf/2S/wBZAl9rDOjkbZMA49HCkKtouNJJeqN66SRMkB9 X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-31--949747287 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Ah, well I'll check back in a week then. But for the record, what I = meant was that nodetool gossipinfo now has entries like: /10.1.20.201 STATUS:LEFT,50,1364152145790 Where is shows "50" is where the token used to be, and where it still is = on all my live nodes. So it appears to me as if all my assassinated = nodes now have a token of 50. Either way, they don't seem to be bugging = the rest of the cluster anymore, so thanks again. On Mar 21, 2013, at 3:05 PM, Alain RODRIGUEZ wrote: > "(And now all sharing token 50? I dunno where that came from.)" >=20 > Not sure about what you mean. >=20 > "nodetool gossipinfo still shows all the old nodes there" >=20 > They must appear with a "left" or "remove" status. Off the top of my = head, this information will remains 7 days. Not sure about it. >=20 >=20 >=20 >=20 > 2013/3/21 Ben Chobot > Thanks Alain, this seems to have stopped the log messages, even though = nodetool gossipinfo still shows all the old nodes there. (And now all = sharing token 50? I dunno where that came from.) Will they eventually = fall away from the cluster, or are they there for good? >=20 > On Mar 21, 2013, at 11:53 AM, Alain RODRIGUEZ wrote: >=20 >> Using the unsafeAssassinateEndpoint function with old IPs from JMX = should do the trick. >>=20 >> This was already discussed in this mailing list, search using = "unsafeAssassinateEndpoint" as keyword to know all that you need to know = about it. >>=20 >> Hope you'll be ok after that. >>=20 >> Alain >>=20 >>=20 >> 2013/3/21 Ben Chobot >> I've got a 1.1.5 cluster, and a few weeks ago I removed some nodes = from it. (I was trying to upgrade nodes from AWS' large to xlarge, and = for some reason that made sense at the time, it seemed better to double = my nodes and then decommission the smaller ones, rather than to simply = rebuild the existing nodes serially.) >>=20 >> Now the remaining nodes are all frequently logging that the old, = decommissioned nodes are dead and that their old token is being = removed.... which is great, I guess, but why does my cluster know about = them at all? Doing a nodetool removetoken doesn't work, as the dead = nodes don't display in the ring. Is this expected behavior after a = nodetool decommission? Is maybe something cached that I can safely = uncache? >>=20 >=20 >=20 --Apple-Mail-31--949747287 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Ah, = well I'll check back in a week then. But for the record, what I meant = was that nodetool gossipinfo now has entries = like:

/10.1.20.201
  = STATUS:LEFT,50,1364152145790

Where is = shows "50" is where the token used to be, and where it still is on all = my live nodes. So it appears to me as if all my assassinated = nodes now have a token of 50. Either way, they don't seem to be bugging = the rest of the cluster anymore, so thanks = again.

On Mar 21, 2013, at 3:05 PM, = Alain RODRIGUEZ wrote:

"(And now all = sharing token 50? I dunno where that came from.)"

Not sure about = what you mean.

"nodetool = gossipinfo still shows all the old nodes there"

They must appear with a "left" or = "remove" status. Off the = top of my head, this information will remains 7 days. Not sure about = it.




2013/3/21 Ben Chobot <bench@instructure.com>
Thanks Alain, this seems to have = stopped the log messages, even though nodetool gossipinfo still shows = all the old nodes there. (And now all sharing token 50? I dunno where = that came from.) Will they eventually fall away from the cluster, or are = they there for good?

On Mar 21, 2013, at 11:53 AM, Alain = RODRIGUEZ wrote:

Using the unsafeAssassinateEndpoint function with old IPs from JMX = should do the trick.

This was already discussed in this mailing = list, search using "unsafeAssassinateEndpoint= " as keyword to know all that you need to know about it.

Hope you'll be ok after = that.

Alain


2013/3/21 Ben Chobot <bench@instructure.com>
I've got a 1.1.5 cluster, and a few weeks ago I removed some nodes from = it. (I was trying to upgrade nodes from AWS' large to xlarge, and for = some reason that made sense at the time, it seemed better to double my = nodes and then decommission the smaller ones, rather than to simply = rebuild the existing nodes serially.)

Now the remaining nodes are all frequently logging that the old, = decommissioned nodes are dead and that their old token is being = removed.... which is great, I guess, but why does my cluster know about = them at all? Doing a nodetool removetoken doesn't work, as the dead = nodes don't display in the ring. Is this expected behavior after a = nodetool decommission? Is maybe something cached that I can safely = uncache?

=



= --Apple-Mail-31--949747287--