Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A84E411CEB for ; Sun, 1 Jun 2014 04:42:25 +0000 (UTC) Received: (qmail 58142 invoked by uid 500); 1 Jun 2014 04:42:23 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 58102 invoked by uid 500); 1 Jun 2014 04:42:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 58094 invoked by uid 99); 1 Jun 2014 04:42:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jun 2014 04:42:23 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bluethundr@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jun 2014 04:42:19 +0000 Received: by mail-we0-f172.google.com with SMTP id k48so3738840wev.31 for ; Sat, 31 May 2014 21:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=X+jumKQ0Iazog8PZg2828eJzD/wF1yygIGH4yScQBZ8=; b=d34oIeG25myLSB2k8yn9bwuqmU4cjDC4BoHlg/N3eCW4bfjA4ve2U+LQGe2o+uubvL msPLq4ruzrZKlmztH8VVRBG4BOKZq1pOZG2IexkS8xKl5GDsO9kTBPxheb1SxvO9XkzP FTvI18OSxqPAdC3HHMB2Vfpt0VKWVLtBXIHQGBzbqjPMUSRvd9IaZkbtS30GuSZgSxPf Wchp0vPK7HY3RoWtQW68rcgIW54ed4FPAzdppobebGKb1WKUXbUywdVfqjfxQ8OoZQy5 1KrAWIfGF2Cj9v/IpuFVhZYUyNb7cMmh7dqowFYOaqizzD3guav6h5EArbMXPWJaTArk oN/g== MIME-Version: 1.0 X-Received: by 10.180.11.37 with SMTP id n5mr11465569wib.41.1401597718163; Sat, 31 May 2014 21:41:58 -0700 (PDT) Received: by 10.216.77.131 with HTTP; Sat, 31 May 2014 21:41:58 -0700 (PDT) In-Reply-To: References: Date: Sun, 1 Jun 2014 00:41:58 -0400 Message-ID: Subject: Re: decommission dead node without token From: Tim Dunphy To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=001a11c25b481bf89804fabee6a5 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c25b481bf89804fabee6a5 Content-Type: text/plain; charset=UTF-8 > > backslash it \-9485547484 > just guessing. Oh, worth a shot! But I'm heading to bed now. I'll try again with this tomorrow. cya Tim On Sun, Jun 1, 2014 at 12:39 AM, Spencer Brown wrote: > backslash it \-9485547484 > just guessing. > > > On Sun, Jun 1, 2014 at 12:31 AM, Tim Dunphy wrote: > >> nodetool -host 10.10.1.68 removetoken -9208584805646615844 >> >> >> I still get a unix usage message with this syntax, yo. Just a theory >> dude, but I think that the negative sign is confusing bash. >> >> But yeah definitely fixing posts is the priority. Then maybe we can >> tackle my cassandra woes. And then move onto improving the font. Mainly >> cassandra is an issue in that I can't do a repair on the database unless >> all nodes are present with none down. I definitely think there's a way out >> of this jam. I hope that the list has some input here. >> >> But definitely don't worry about JF until you're caught up with work. I >> want to see you keep this job going as much as you can. Like I do with mine >> at NBCU! :) >> >> on fixing posting or nodes? I don't think posting fix should be too >>> hard. I'm mainly just late with my primary job in fixing something which >>> should be easy. It's an AJAX sort of thing. Have you dealt with that... >>> it's hard. It means Asynchronous Javascript and XML. And there's an HTML >>> element too. And in this case there is some JQuery. And when you combine >>> all that it becomes a pain in the ass. >> >> >> Never really dealt with AJAX. Always meant to get into it tho. I hear it >> can be powerful! But definitely good luck with that! And I hope you can >> indeed get to do some JF stuff this weekend. But don't stress about it if >> you're stuck with work. >> >> Tim >> >> >> >> >> >> On Sun, Jun 1, 2014 at 12:21 AM, Spencer Brown >> wrote: >> >>> nodetool -host 10.10.1.68 removetoken -9208584805646615844 >>> >>> >>> On Sun, Jun 1, 2014 at 12:17 AM, Tim Dunphy >>> wrote: >>> >>>> sorry for the barrage of questions.... but I'm having trouble with the >>>> nodetool remove phase. >>>> >>>> [root@cassandra01 ~]# nodetool removetoken -h 10.10.1.68 >>>> -9208584805646615844 >>>> >>>> Gives me a unix usage message. As in: >>>> >>>> [root@cassandra01 ~]# nodetool removetoken -h 107.170.178.68 >>>> -9208584805646615844 | head -5 >>>> Unrecognized option: -9208584805646615844 >>>> usage: java org.apache.cassandra.tools.NodeCmd --host >>>> >>>> -a,--include-all-sstables includes sstables that are already on the >>>> most recent version during upgradesstables >>>> -c,--compact print histograms in a more compact format >>>> >>>> I think the problem is that the bash environment is getting confused by >>>> the minus sign in front of the token. But that is the token and the minus >>>> sign is integral to it. I've even tried quoting it with both single quotes >>>> and double quotes to no avail. I still get the same usage message. >>>> >>>> Can someone please help with the right syntax here? >>>> >>>> Thanks >>>> Tim >>>> >>>> >>>> On Sun, Jun 1, 2014 at 12:01 AM, Tim Dunphy >>>> wrote: >>>> >>>>> Just wondering.. might I be able to retried the 'lost' intital_token >>>>> of the node I want to remove by going: >>>>> >>>>> [root@beta-new:~] #nodetool ring | grep 10.10.1.102 | head -1 >>>>> 107.170.175.102 rack1 Down Normal 123.41 KB 32.19% >>>>> -9208584805646615844 >>>>> >>>>> Then >>>>> >>>>> 1) place that token in the cassandra.yaml of a new node with a new IP >>>>> as -9208584805646615844 -1 >>>>> 2) set auto_bootstrap in the cassandra.yaml >>>>> 3) once the node boots run 'nodetool removetoken -9208584805646615844' >>>>> 4) run nodetool cleanup >>>>> >>>>> Would this be the way to get out of this bind? Just want to be sure I >>>>> understand the process. >>>>> >>>>> Thanks >>>>> Tim >>>>> >>>>> >>>>> On Sat, May 31, 2014 at 10:38 PM, Tim Dunphy >>>>> wrote: >>>>> >>>>>> hey guys, >>>>>> >>>>>> I have two dead nodes in my ring. >>>>>> >>>>>> Status=Up/Down >>>>>> >>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>> >>>>>> -- Address Load Tokens Owns Host ID >>>>>> Rack >>>>>> >>>>>> DN 10.10.1.102 123.41 KB 256 32.2% >>>>>> ddcafc75-24ed-4c6a-99bb-afe3dd551a9c rack1 >>>>>> >>>>>> UN 10.10.1.94 170.81 KB 256 35.4% >>>>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e rack1 >>>>>> >>>>>> DN 10..10.10.64 ? 256 32.5% >>>>>> f2a48fc7-a362-43f5-9061-4bb3739fdeaf rack1 >>>>>> >>>>>> >>>>>> Unfortunately I am early in my learning curve and failed to preserve >>>>>> the initial_token for each of the nodes that were taken out of service. As >>>>>> things stand now I cannot do a nodetool repair on the main keyspace because >>>>>> of the dead nodes. >>>>>> >>>>>> Is there any way to decommission the dead nodes if you no longer have >>>>>> their initial_token values? >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Tim >>>>>> >>>>>> -- >>>>>> GPG me!! >>>>>> >>>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> GPG me!! >>>>> >>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>>>> >>>>> >>>> >>>> >>>> -- >>>> GPG me!! >>>> >>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>>> >>>> >>> >> >> >> -- >> GPG me!! >> >> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >> >> > -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B --001a11c25b481bf89804fabee6a5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
backslash it =C2=A0\-9485547484
just guessing.

Oh, worth a shot! But= I'm heading to bed now. I'll try again with this tomorrow.=C2=A0

cya

Tim=C2=A0
<= div class=3D"gmail_extra">

On Sun, Jun 1, 2014 at 12:39 AM, Spencer= Brown <lilspenny@gmail.com> wrote:
backslash it =C2=A0\-9485547484
just guessing.

=
On Sun, Jun 1, 2014 at 12:31 AM, Tim Dunphy = <bluethundr@gmail.com> wrote:
nodetool -host = 10.10.1.68=C2=A0removetoken=C2=A0-9208584805646615844

I still get a unix usage message with this syntax, yo. Just = a theory dude, but I think that the negative sign is confusing bash.
But yeah definitely fixing posts is the priority. Then maybe w= e can tackle my cassandra woes. And then move onto improving the font. =C2= =A0Mainly cassandra is an issue in that I can't do a repair on the data= base unless all nodes are present with none down. I definitely think there&= #39;s a way out of this jam. I hope that the list has some input here.

=C2=A0But definitely don't worry about JF until you= 're caught up with work. I want to see you keep this job going as much = as you can. Like I do with mine at NBCU! :)=C2=A0

on fixing posti= ng or nodes? =C2=A0I don't think posting fix should be too hard. =C2=A0= I'm mainly just late with my primary job in fixing something which shou= ld be easy. =C2=A0It's an AJAX sort of thing. =C2=A0Have you dealt with= that... =C2=A0it's hard. =C2=A0It means Asynchronous Javascript and XM= L. =C2=A0And there's an HTML element too. =C2=A0And in this case there = is some JQuery. =C2=A0And when you combine all that it becomes a pain in th= e ass.

Never really dealt with AJAX. Always meant to get into = it tho. I hear it can be powerful! But definitely good luck with that! And = I hope you can indeed get to do some JF stuff this weekend. But don't s= tress about it if you're stuck with work.

Tim
=C2=A0


<= /div>


On= Sun, Jun 1, 2014 at 12:21 AM, Spencer Brown <lilspenny@gmail.com&g= t; wrote:
nodetool -host 10.10.1.68=C2=A0removetoken=C2=A0-920858480= 5646615844


On Sun, Jun 1, 2014 at 12:17 AM, Tim Dunphy <bluethundr@gmail.com> wrote:
sorry for the barrage of qu= estions.... but I'm having trouble with the nodetool remove phase.=C2= =A0

[root@cassandra01 ~]# nodetool removetoken -h 10.10.1.68 -920858= 4805646615844

Gives me a unix usage message. As in:

[root@cassandra01 ~]# nodetool removetoken -h 107.170.178.68 -92085= 84805646615844 | head -5
Unrecognized option: -920858480564661584= 4
usage: java org.apache.cassandra.tools.NodeCmd --host <arg> <= command>

=C2=A0-a,--include-all-sstables =C2=A0= includes sstables that are already on the
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0most recent version during upgradesstables
=C2=A0-c,--compact =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0print histograms in a more compact format

<= div>I think the problem is that the bash environment is getting confused by= the minus sign in front of the token. But that is the token and the minus = sign is integral to it. I've even tried quoting it with both single quo= tes and double quotes to no avail. I still get the same usage message.

Can someone please help with the right syntax here?

Thanks
Tim


On Sun, Jun 1, 2014 at 12:01 AM, T= im Dunphy <bluethundr@gmail.com> wrote:
Just wondering.. might I be= able to retried the 'lost' intital_token of the node I want to rem= ove by going:

[root@beta-new:~] #nodetool ring | grep 10.10.1.102 | h= ead -1
107.170.175.102 =C2=A0rack1 =C2=A0 =C2=A0 =C2=A0 Down =C2=A0 Normal = =C2=A0123.41 KB =C2=A0 =C2=A0 =C2=A0 32.19% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0-9208584805646615844

Then

1) place that token in the cassandra.yaml of a new = node with a new IP as =C2=A0-9208584805646615844 -1=C2=A0
2) =C2=A0set auto_bootstrap in the cassandra.yaml
3) once th= e node boots run 'nodetool removetoken -9208584805646615844'
<= div>4) run nodetool cleanup

Would this be the way = to get out of this bind? Just want to be sure I understand the process.

Thanks
Tim


On Sat, May 31, 2014 at 10:38 PM, Tim Dunphy <bluethu= ndr@gmail.com> wrote:
hey guys,
=C2=A0I have two dead nodes in my ring.=C2=A0
<= br>

Status=3DUp/Down

|/ State=3DNormal/Leaving/Joining/Moving

--=C2=A0 Address=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Load =C2=A0 =C2=A0 = =C2=A0 Tokens=C2=A0 Owns =C2=A0 Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Rack<= /p>

DN=C2=A0 10.10.1.102=C2=A0 123.41 KB=C2=A0 256 =C2=A0 =C2=A0 32.2%=C2=A0= ddcafc75-24ed-4c6a-99bb-afe3dd551a9c=C2=A0 rack1

UN=C2=A0 10.10.1.94 =C2=A0 170.81 KB=C2=A0 256 =C2=A0 =C2=A0 35.4%=C2=A0= fd2f76ae-8dcf-4e93-a37f-bf1e9088696e=C2=A0 rack1

DN=C2=A0 10..10.10.64 =C2=A0 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256 =C2= =A0 =C2=A0 32.5%=C2=A0 f2a48fc7-a362-43f5-9061-4bb3739fdeaf=C2=A0 rack1

=


Unfortunately I am early in my learning curve and failed to p= reserve the initial_token for each of the nodes that were taken out of serv= ice. As things stand now I cannot do a nodetool repair on the main keyspace= because of the dead nodes.=C2=A0

Is there any way to decommission the dead nodes if you no longer have th= eir initial_token values?


Thanks

Tim


--
GPG me!!

gpg --keyserver pool.sks-keyserve= rs.net --recv-keys F186197B




--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197= B




--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv= -keys F186197B


<= /span>
=


--
GPG me!!

gpg --keyserver= pool.sks-keys= ervers.net --recv-keys F186197B



--
= GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

--001a11c25b481bf89804fabee6a5--