Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9914A11CE9 for ; Sun, 1 Jun 2014 04:40:20 +0000 (UTC) Received: (qmail 55263 invoked by uid 500); 1 Jun 2014 04:40:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 55223 invoked by uid 500); 1 Jun 2014 04:40:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 55215 invoked by uid 99); 1 Jun 2014 04:40:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jun 2014 04:40:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lilspenny@gmail.com designates 209.85.213.53 as permitted sender) Received: from [209.85.213.53] (HELO mail-yh0-f53.google.com) (209.85.213.53) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jun 2014 04:40:15 +0000 Received: by mail-yh0-f53.google.com with SMTP id i57so2884484yha.12 for ; Sat, 31 May 2014 21:39:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0dKnxoKtVazZu3lVUuq2bqmaZNDV/064jaWQqNpHyCU=; b=rQUtklwgAMZRbqEkWZxLKbr5yv8MJag010OnbtJR7qXMrFPHrxcOBWqIvou+wu+NGB bDZA5jEfw/iM2VySyYw5SeMd7l/IDQ6iHx4fahachIPD7uA1n5so919T5XP+v4HrIO+e cfut3OOGUwefQQEicTa1TvHL3ZBva0T8c8fQ7nyI0GNNRHgZE5zVgy+xyKloqZUDbxr6 obbkIL0E6mMpot7g4nW4HvO+GzBul59VzVfS4X3J41+jeCH7Kl3QFVGJqRvA3dPEUq3X eybS9LEklRF0FKUlC76QcnRHFAlQ4gUBTGMnbE6um+SaPOxgmBhq3nY5VAtlnhhkN8z5 WjVA== MIME-Version: 1.0 X-Received: by 10.236.150.205 with SMTP id z53mr38485161yhj.75.1401597591428; Sat, 31 May 2014 21:39:51 -0700 (PDT) Received: by 10.170.120.215 with HTTP; Sat, 31 May 2014 21:39:51 -0700 (PDT) In-Reply-To: References: Date: Sun, 1 Jun 2014 00:39:51 -0400 Message-ID: Subject: Re: decommission dead node without token From: Spencer Brown To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf303a2d618e233c04fabede6d X-Virus-Checked: Checked by ClamAV on apache.org --20cf303a2d618e233c04fabede6d Content-Type: text/plain; charset=UTF-8 backslash it \-9485547484 just guessing. On Sun, Jun 1, 2014 at 12:31 AM, Tim Dunphy wrote: > nodetool -host 10.10.1.68 removetoken -9208584805646615844 > > > I still get a unix usage message with this syntax, yo. Just a theory dude, > but I think that the negative sign is confusing bash. > > But yeah definitely fixing posts is the priority. Then maybe we can tackle > my cassandra woes. And then move onto improving the font. Mainly cassandra > is an issue in that I can't do a repair on the database unless all nodes > are present with none down. I definitely think there's a way out of this > jam. I hope that the list has some input here. > > But definitely don't worry about JF until you're caught up with work. I > want to see you keep this job going as much as you can. Like I do with mine > at NBCU! :) > > on fixing posting or nodes? I don't think posting fix should be too hard. >> I'm mainly just late with my primary job in fixing something which should >> be easy. It's an AJAX sort of thing. Have you dealt with that... it's >> hard. It means Asynchronous Javascript and XML. And there's an HTML >> element too. And in this case there is some JQuery. And when you combine >> all that it becomes a pain in the ass. > > > Never really dealt with AJAX. Always meant to get into it tho. I hear it > can be powerful! But definitely good luck with that! And I hope you can > indeed get to do some JF stuff this weekend. But don't stress about it if > you're stuck with work. > > Tim > > > > > > On Sun, Jun 1, 2014 at 12:21 AM, Spencer Brown > wrote: > >> nodetool -host 10.10.1.68 removetoken -9208584805646615844 >> >> >> On Sun, Jun 1, 2014 at 12:17 AM, Tim Dunphy wrote: >> >>> sorry for the barrage of questions.... but I'm having trouble with the >>> nodetool remove phase. >>> >>> [root@cassandra01 ~]# nodetool removetoken -h 10.10.1.68 >>> -9208584805646615844 >>> >>> Gives me a unix usage message. As in: >>> >>> [root@cassandra01 ~]# nodetool removetoken -h 107.170.178.68 >>> -9208584805646615844 | head -5 >>> Unrecognized option: -9208584805646615844 >>> usage: java org.apache.cassandra.tools.NodeCmd --host >>> >>> -a,--include-all-sstables includes sstables that are already on the >>> most recent version during upgradesstables >>> -c,--compact print histograms in a more compact format >>> >>> I think the problem is that the bash environment is getting confused by >>> the minus sign in front of the token. But that is the token and the minus >>> sign is integral to it. I've even tried quoting it with both single quotes >>> and double quotes to no avail. I still get the same usage message. >>> >>> Can someone please help with the right syntax here? >>> >>> Thanks >>> Tim >>> >>> >>> On Sun, Jun 1, 2014 at 12:01 AM, Tim Dunphy >>> wrote: >>> >>>> Just wondering.. might I be able to retried the 'lost' intital_token of >>>> the node I want to remove by going: >>>> >>>> [root@beta-new:~] #nodetool ring | grep 10.10.1.102 | head -1 >>>> 107.170.175.102 rack1 Down Normal 123.41 KB 32.19% >>>> -9208584805646615844 >>>> >>>> Then >>>> >>>> 1) place that token in the cassandra.yaml of a new node with a new IP >>>> as -9208584805646615844 -1 >>>> 2) set auto_bootstrap in the cassandra.yaml >>>> 3) once the node boots run 'nodetool removetoken -9208584805646615844' >>>> 4) run nodetool cleanup >>>> >>>> Would this be the way to get out of this bind? Just want to be sure I >>>> understand the process. >>>> >>>> Thanks >>>> Tim >>>> >>>> >>>> On Sat, May 31, 2014 at 10:38 PM, Tim Dunphy >>>> wrote: >>>> >>>>> hey guys, >>>>> >>>>> I have two dead nodes in my ring. >>>>> >>>>> Status=Up/Down >>>>> >>>>> |/ State=Normal/Leaving/Joining/Moving >>>>> >>>>> -- Address Load Tokens Owns Host ID >>>>> Rack >>>>> >>>>> DN 10.10.1.102 123.41 KB 256 32.2% >>>>> ddcafc75-24ed-4c6a-99bb-afe3dd551a9c rack1 >>>>> >>>>> UN 10.10.1.94 170.81 KB 256 35.4% >>>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e rack1 >>>>> >>>>> DN 10..10.10.64 ? 256 32.5% >>>>> f2a48fc7-a362-43f5-9061-4bb3739fdeaf rack1 >>>>> >>>>> >>>>> Unfortunately I am early in my learning curve and failed to preserve >>>>> the initial_token for each of the nodes that were taken out of service. As >>>>> things stand now I cannot do a nodetool repair on the main keyspace because >>>>> of the dead nodes. >>>>> >>>>> Is there any way to decommission the dead nodes if you no longer have >>>>> their initial_token values? >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Tim >>>>> >>>>> -- >>>>> GPG me!! >>>>> >>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>>>> >>>>> >>>> >>>> >>>> -- >>>> GPG me!! >>>> >>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>>> >>>> >>> >>> >>> -- >>> GPG me!! >>> >>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>> >>> >> > > > -- > GPG me!! > > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B > > --20cf303a2d618e233c04fabede6d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
backslash it =C2=A0\-9485547484
just guessing.


On Sun, J= un 1, 2014 at 12:31 AM, Tim Dunphy <bluethundr@gmail.com>= wrote:
nodetool -host = 10.10.1.68=C2=A0removetoken=C2=A0-9208584805646615844

I still get a unix usage message with this syntax, yo. Just = a theory dude, but I think that the negative sign is confusing bash.
But yeah definitely fixing posts is the priority. Then maybe w= e can tackle my cassandra woes. And then move onto improving the font. =C2= =A0Mainly cassandra is an issue in that I can't do a repair on the data= base unless all nodes are present with none down. I definitely think there&= #39;s a way out of this jam. I hope that the list has some input here.

=C2=A0But definitely don't worry about JF until you= 're caught up with work. I want to see you keep this job going as much = as you can. Like I do with mine at NBCU! :)=C2=A0

on fixing posti= ng or nodes? =C2=A0I don't think posting fix should be too hard. =C2=A0= I'm mainly just late with my primary job in fixing something which shou= ld be easy. =C2=A0It's an AJAX sort of thing. =C2=A0Have you dealt with= that... =C2=A0it's hard. =C2=A0It means Asynchronous Javascript and XM= L. =C2=A0And there's an HTML element too. =C2=A0And in this case there = is some JQuery. =C2=A0And when you combine all that it becomes a pain in th= e ass.

Never really dealt with AJAX. Always meant to get into = it tho. I hear it can be powerful! But definitely good luck with that! And = I hope you can indeed get to do some JF stuff this weekend. But don't s= tress about it if you're stuck with work.

Tim
=C2=A0


<= /div>


On= Sun, Jun 1, 2014 at 12:21 AM, Spencer Brown <lilspenny@gmail.com&g= t; wrote:
nodetool -host 10.10.1.68=C2=A0removetoken=C2=A0-920858480= 5646615844


On Sun, Jun 1, 2014 at 12:17 AM, Tim Dunphy <bluethundr@gmail.com> wrote:
sorry for the barrage of qu= estions.... but I'm having trouble with the nodetool remove phase.=C2= =A0

[root@cassandra01 ~]# nodetool removetoken -h 10.10.1.68 -920858= 4805646615844

Gives me a unix usage message. As in:

[root@cassandra01 ~]# nodetool removetoken -h 107.170.178.68 -92085= 84805646615844 | head -5
Unrecognized option: -920858480564661584= 4
usage: java org.apache.cassandra.tools.NodeCmd --host <arg> <= command>

=C2=A0-a,--include-all-sstables =C2=A0= includes sstables that are already on the
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0most recent version during upgradesstables
=C2=A0-c,--compact =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0print histograms in a more compact format

<= div>I think the problem is that the bash environment is getting confused by= the minus sign in front of the token. But that is the token and the minus = sign is integral to it. I've even tried quoting it with both single quo= tes and double quotes to no avail. I still get the same usage message.

Can someone please help with the right syntax here?

Thanks
Tim


On Sun, Jun 1, 2014 at 12:01 AM, T= im Dunphy <bluethundr@gmail.com> wrote:
Just wondering.. might I be= able to retried the 'lost' intital_token of the node I want to rem= ove by going:

[root@beta-new:~] #nodetool ring | grep 10.10.1.102 | h= ead -1
107.170.175.102 =C2=A0rack1 =C2=A0 =C2=A0 =C2=A0 Down =C2=A0 Normal = =C2=A0123.41 KB =C2=A0 =C2=A0 =C2=A0 32.19% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0-9208584805646615844

Then

1) place that token in the cassandra.yaml of a new = node with a new IP as =C2=A0-9208584805646615844 -1=C2=A0
2) =C2=A0set auto_bootstrap in the cassandra.yaml
3) once th= e node boots run 'nodetool removetoken -9208584805646615844'
<= div>4) run nodetool cleanup

Would this be the way = to get out of this bind? Just want to be sure I understand the process.

Thanks
Tim


On Sat, May 31, 2014 at 10:38 PM, Tim Dunphy <bluethu= ndr@gmail.com> wrote:
hey guys,
=C2=A0I have two dead nodes in my ring.=C2=A0
<= br>

Status=3DUp/Down

|/ State=3DNormal/Leaving/Joining/Moving

--=C2=A0 Address=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Load =C2=A0 =C2=A0 = =C2=A0 Tokens=C2=A0 Owns =C2=A0 Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Rack<= /p>

DN=C2=A0 10.10.1.102=C2=A0 123.41 KB=C2=A0 256 =C2=A0 =C2=A0 32.2%=C2=A0= ddcafc75-24ed-4c6a-99bb-afe3dd551a9c=C2=A0 rack1

UN=C2=A0 10.10.1.94 =C2=A0 170.81 KB=C2=A0 256 =C2=A0 =C2=A0 35.4%=C2=A0= fd2f76ae-8dcf-4e93-a37f-bf1e9088696e=C2=A0 rack1

DN=C2=A0 10..10.10.64 =C2=A0 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256 =C2= =A0 =C2=A0 32.5%=C2=A0 f2a48fc7-a362-43f5-9061-4bb3739fdeaf=C2=A0 rack1

=


Unfortunately I am early in my learning curve and failed to p= reserve the initial_token for each of the nodes that were taken out of serv= ice. As things stand now I cannot do a nodetool repair on the main keyspace= because of the dead nodes.=C2=A0

Is there any way to decommission the dead nodes if you no longer have th= eir initial_token values?


Thanks

Tim


--
GPG me!!

gpg --keyserver pool.sks-keyserve= rs.net --recv-keys F186197B




--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197= B




--
GPG me!!

gpg --keyserver pool.sks-keyserve= rs.net --recv-keys F186197B





--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
--20cf303a2d618e233c04fabede6d--