Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3ED24200BC3 for ; Fri, 18 Nov 2016 22:07:54 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3D4FF160B04; Fri, 18 Nov 2016 21:07:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 613DA160B03 for ; Fri, 18 Nov 2016 22:07:53 +0100 (CET) Received: (qmail 954 invoked by uid 500); 18 Nov 2016 21:07:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 944 invoked by uid 99); 18 Nov 2016 21:07:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Nov 2016 21:07:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 58553185F3D for ; Fri, 18 Nov 2016 21:07:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.091 X-Spam-Level: *** X-Spam-Status: No, score=3.091 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, HTML_OBFUSCATE_10_20=1.162, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id fvaQ6cNGz9t1 for ; Fri, 18 Nov 2016 21:07:49 +0000 (UTC) Received: from mail-it0-f46.google.com (mail-it0-f46.google.com [209.85.214.46]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8269B5FC5F for ; Fri, 18 Nov 2016 21:07:49 +0000 (UTC) Received: by mail-it0-f46.google.com with SMTP id j191so53166021ita.1 for ; Fri, 18 Nov 2016 13:07:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Rg7VaNKGXBJsUDIHaa7E73zNfcj1lb0gnizvmSS1bzw=; b=XhT0MDs2b2f0058Tb+mEmNzcK+P4s1HjLBpQhuDKezk0Q6fNtnS3bvH8O3GE17u3l+ ql9G1q88p+S9ed1QmnJQpLacoarnfFuVye5bfYHKdKrni3eQSmRzeHhDiWB+YNO1Tebb 1CirDmFhwFz6p8mbVzYlgTPzToYSgBnZfkXzYf9yvqvOicPxDrTfQYNZzCMNhaNbpKzZ lFyiL3BFguGzV+CdsdCCViBQ+TC+59TTxMLmJIVYsZF02fFnQQeP1nlOKlU4fi3GSdPn SRtz5jNVZqatdBJIktlHPEo3a0SqnFfkR7fRPSXr7jq9szMrr56xAjcOYJGdv2oCUM73 zyOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Rg7VaNKGXBJsUDIHaa7E73zNfcj1lb0gnizvmSS1bzw=; b=ZBfQ2f1e/YdrS3ar46G5imgY80UG1XSACZQ5/pn0q3u1aM6Egd/rei4B3Dhupxnp6i zmuMKp1pBMMXk7g6IoDwr53e7FmyID0K++JnSfXLR0P/BSQCWVFZaeboHasodILanOxe sErTH26aJ40wjgwG1X1xlJbMCOneqYbUZaN0QwHN3pXveXW/oSxcvxO8dkasCep4qshX 1hfEYwqAyKbKhKJBGW+eaWJZ36gsRvLI7CwE+sL63lJw/V/4Xeawep5/wN2al8IRk0o8 pTtYfmUNQTcIseoud9PaWo0wpc5cEepiO1HhjjmWZRtoVFm8l19zD8p2WEkgzyutqvBT AFVw== X-Gm-Message-State: AKaTC00nmiatkQs+FjN4MnAv+cwCRppiIqDf1CzOTFzbLszC5hfYg7bkqV1rZU25KPIIkV/l4yRfOl1CTL2UNg== X-Received: by 10.36.233.69 with SMTP id f66mr398329ith.83.1479503268048; Fri, 18 Nov 2016 13:07:48 -0800 (PST) MIME-Version: 1.0 Received: by 10.36.98.15 with HTTP; Fri, 18 Nov 2016 13:07:27 -0800 (PST) In-Reply-To: References: From: Dikang Gu Date: Fri, 18 Nov 2016 13:07:27 -0800 Message-ID: Subject: Re: Node replacement failed in 2.2 To: cassandra Content-Type: multipart/alternative; boundary=94eb2c11718cbc75df054199b260 archived-at: Fri, 18 Nov 2016 21:07:54 -0000 --94eb2c11718cbc75df054199b260 Content-Type: text/plain; charset=UTF-8 Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows "TOKENS: not present", on all live nodes. It means tokens are missing, right? What would cause this? Thanks. Dikang. On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta wrote: > What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091: > face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is > it only missing on this node or other nodes as well? > > 2016-11-18 17:02 GMT-02:00 Dikang Gu : > >> Hi, I encountered couple times that I could not replace a down node due >> to error: >> >> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException) >> encountered during startup: Could not find tokens for >> /2401:db00:2130:4091:face:0:13:0 to replace >> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered >> during startup >> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find >> tokens for /2401:db00:2130:4091:face:0:13:0 to replace >> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.S >> torageService.prepareReplacementInfo(StorageService.java:525) >> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >> 160315.c29948b] >> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >> torageService.prepareToJoin(StorageService.java:760) >> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >> 160315.c29948b] >> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >> torageService.initServer(StorageService.java:693) >> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >> 160315.c29948b] >> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >> torageService.initServer(StorageService.java:585) >> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >> 160315.c29948b] >> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.C >> assandraDaemon.setup(CassandraDaemon.java:300) >> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >> 60315.c29948b] >> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C >> assandraDaemon.activate(CassandraDaemon.java:516) >> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >> 60315.c29948b] >> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C >> assandraDaemon.main(CassandraDaemon.java:625) >> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >> 60315.c29948b] >> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]: >> Announcing shutdown >> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]: >> Waiting for messaging service to quiesce >> 2016-11-17_19:34:00.71066 INFO 19:34:00 [ACCEPT-/2401:db00:2130:4091:face:0:13:0]: >> MessagingService has terminated the accept() thread >> >> Did not find a relevant ticket for this, is anyone aware of this? >> >> Thanks! >> >> -- >> Dikang >> >> > -- Dikang --94eb2c11718cbc75df054199b260 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Paulo, the tokens field for=C2=A02401:db00:2130:4091:face:= 0:13:0 shows "TOKENS: not present", on all live nodes. It means t= okens are missing, right? What would cause this?

Thanks.=
Dikang.

On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <paulori= cardomg@gmail.com> wrote:
<= div dir=3D"ltr">What does nodetool gossipinfo shows for endpoint=C2=A0/2401:db00:2130:4091:face:0:13:0 ? Does it contain the TOKENS attribute? If i= t's missing, is it only missing on this node or other nodes as well?

2016-11-18 17:02 GMT-02:00 Dikang Gu <di= kang85@gmail.com>:
Hi, I encountered couple times that I could not replace a down nod= e due to error:

2016-11-17_19:33:58.70075 Exception= (java.lang.RuntimeException) encountered during startup: Could not find to= kens for /2401:db00:2130:4091:face:0:13:0 to replace
2016-11= -17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered during star= tup
2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could n= ot find tokens for /2401:db00:2130:4091:face:0:13:0 to replace
2016-11-17_19:33:58.70491 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.s= ervice.StorageService.prepareReplacementInfo(StorageService.java:= 525) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20= 160315.c29948b]
2016-11-17_19:33:58.70492 =C2=A0 =C2=A0 =C2= =A0 at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:760) ~[apache-cassandra-2.2.5+git20160315.c29948b.= jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70492 =C2= =A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.StorageService.initS= erver(StorageService.java:693) ~[apache-cassandra-2.2.5+git201603= 15.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:5= 8.70492 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.StorageSe= rvice.initServer(StorageService.java:585) ~[apache-cassandra-2.2.5+git= 20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11= -17_19:33:58.70492 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.= 2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
= 2016-11-17_19:33:58.70493 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.serv= ice.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-ca= ssandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]<= /div>
2016-11-17_19:33:58.70493 =C2=A0 =C2=A0 =C2=A0 at org.apache.cass= andra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apa= che-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29= 948b]
2016-11-17_19:33:58.70649 INFO =C2=A019:33:58 [StorageServi= ceShutdownHook]: Announcing shutdown
2016-11-17_19:34:00.70967 IN= FO =C2=A019:34:00 [StorageServiceShutdownHook]: Waiting for messaging servi= ce to quiesce
2016-11-17_19:34:00.71066 INFO =C2=A019:34:00 [ACCE= PT-/2401:db00:2130:4091:face:0:13:0]: MessagingService has terminated = the accept() thread

Did not find a relevant ticket= for this, is anyone aware of this?

Thanks!
<= span class=3D"m_-7338533314632750374HOEnZb">

--
Dikang





--
=
Dikang

--94eb2c11718cbc75df054199b260--