Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9F3F3200BF4 for ; Fri, 6 Jan 2017 20:37:49 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9DD1E160B39; Fri, 6 Jan 2017 19:37:49 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9D2B4160B37 for ; Fri, 6 Jan 2017 20:37:48 +0100 (CET) Received: (qmail 51364 invoked by uid 500); 6 Jan 2017 19:37:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51349 invoked by uid 99); 6 Jan 2017 19:37:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Jan 2017 19:37:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 703B1C023E for ; Fri, 6 Jan 2017 19:37:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.62 X-Spam-Level: X-Spam-Status: No, score=-0.62 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLYTO_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id V7Vvz1NqKxh6 for ; Fri, 6 Jan 2017 19:37:44 +0000 (UTC) Received: from nm19-vm5.bullet.mail.ne1.yahoo.com (nm19-vm5.bullet.mail.ne1.yahoo.com [98.138.91.241]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 929075F477 for ; Fri, 6 Jan 2017 19:37:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1483731458; bh=cIAfizbSrgxMO/Cn+yL+YL4YAY7+yw5miAPFtOT8jUA=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=Hw0GGOiaYp2AJZJ9sL757XKgCbg7wfuXNmTQDYA43sODzIhp+LTSpibw23e2iIbiCDqIjtODg0z7rUjPZjmmfcEyqhC7U7TBWD2PsS01XpLeKMlfm/VYTkll7gxW+OB4E+uDBMEbkCxltyOZziqrSTAWvmkbRIp3B9pnPollxHOJmQ9bEI9mgzgmbeZdwcBN1Fo0jMtq2vO/SEEMIiSMqQ3T/RRivy708rJgGGtCBBmxYmnn2MIROHFBQF14fqU6MW3vO9fwGExW8GU4by33SldwujUlyOT54hJIKbSLq1oNzBQrpamlpEGEqJKAR+24Bv3qo5ZB87m9vv//aT+O/g== Received: from [98.138.100.112] by nm19.bullet.mail.ne1.yahoo.com with NNFMP; 06 Jan 2017 19:37:38 -0000 Received: from [98.138.89.168] by tm103.bullet.mail.ne1.yahoo.com with NNFMP; 06 Jan 2017 19:37:37 -0000 Received: from [127.0.0.1] by omp1024.mail.ne1.yahoo.com with NNFMP; 06 Jan 2017 19:37:37 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 973636.74476.bm@omp1024.mail.ne1.yahoo.com X-YMail-OSG: EAMhy7AVM1kCYPbAWng9Xx6V59WyL53332._p6ogxgYF9Dr2kNsy262IQNcPQpt yhVRI3dhokhbAlXC3srQe3GO4lyXTlHLNt_MlbCVL2vMM2qspPLHKOOBGIUhPO_WmPtDmk3A.Bd6 hE_HOGxMD3SmUqwGOb72YTLdhtDeHA2H_I3_yzM_A1WJlfq5LEhCpqTVLFh.2KEQkMrOaSjmzfTl WbLvj.u4pGsYQg1WJnigCTzRcx9O1_CjWHnWGiBEMEChuAFTd_aV73eiU0ByAzYD24JC5bI7QqS5 49ApA1lSoLDyctE8.RwdT6I9VxDh.ni3AHcnD_ySQGkeRahKb3ifJLPzCB4JQyprwOgZgRTYH.BB OpRf3YuKQub7aMtO4XP0e.v3Xhv3HEMofEUwFzXfjZZlpIxaDxe0cAtWmAofgTvgSWK6ac640fgh pG1xSrt2ARxY4kmzw0xbjNEYQMKCqbKCnJ02a_ZRInoey6.IG9zShiYQEKCKUHLtm5N4UPvN7fBq dRhbUtmSvn8dfYY00HztJlrg1I1OzdfGwaOUmILc7PyfFRqsnij3Zagm.MQ9MJq5PHPW3pFUNrH4 QjGU- Received: from jws200013.mail.ne1.yahoo.com by sendmailws159.mail.ne1.yahoo.com; Fri, 06 Jan 2017 19:37:37 +0000; 1483731457.594 Date: Fri, 6 Jan 2017 19:37:37 +0000 (UTC) From: Sotirios Delimanolis Reply-To: Sotirios Delimanolis To: User Message-ID: <1959545284.1303559.1483731457369@mail.yahoo.com> Subject: Logs appear to contradict themselves during bootstrap steps MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1303558_270961676.1483731457367" References: <1959545284.1303559.1483731457369.ref@mail.yahoo.com> archived-at: Fri, 06 Jan 2017 19:37:49 -0000 ------=_Part_1303558_270961676.1483731457367 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable We had a node go down in our cluster and its disk had to be wiped. During t= hat time, all nodes in the cluster have restarted at least once. We want to add the bad node back to the ring. It has the same IP/hostname. = I follow the steps=C2=A0here=C2=A0for "Adding nodes to an existing cluster.= " When the process is started up, it reports A node with address /
already exists, cancelling join. U= se cassandra.replace_address if you want to replace this node. I found this error message in the StorageService using the Gossiper instanc= e to look up the node's state. Apparently, the node knows about it. So I fo= llowed the instructions and added the cassandra.replace_address system prop= erty and restarted the process. But it reports Cannot replace_address /
because it doesn't exist in gossip So which one is it? Does the ring know about it or not? Running "nodetool r= ing" does show it on all other nodes. I've seen=C2=A0CASSANDRA-8138=C2=A0andthe conditions are the same, but I ca= n't understand why it thinks it's not part of gossip. What's the difference= between the gossip check used to make this determination and the gossip ch= eck used for the first error message?=C2=A0Can someone explain? I've since retrieved the node's id and used it to "nodetool removenode". Af= ter rebalancing, I added the node back and "nodetool cleaned" up. Everythin= g's up and running, but I'd like to understand what Cassandra was doing. ------=_Part_1303558_270961676.1483731457367 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
We had a node go down in our cluster and it= s disk had to be wiped. During that time, all nodes in the cluster have res= tarted at least once.

We w= ant to add the bad node back to the ring. It has the same IP/hostname. I fo= llow the steps here for "Adding nodes to an = existing cluster."

When th= e process is started up, it reports

A node with addr= ess <hostname>/<address> already exists, cancelling join. Use c= assandra.replace_address if you=20 want to replace this node.

I found this error message in the StorageServi= ce using the Gossiper instance to look up the node's state. Apparen= tly, the node knows about it. So I followed the instructions and added the = cassandra.replace_address syst= em property and restarted the process.

= But it reports

=
Cannot replace_address /<address> because it doesn't exist in= gossip

So which one is it? Does the ring know about it or not? Running "nodetool ring" does show it on a= ll other nodes.

I've seen CASSANDRA-8138 andthe conditions are the same, b= ut I can't understand why it thinks it's not part of gossip. What's the dif= ference between the gossip check used to make this determination and the go= ssip check used for the first error message? Can someone explain?

I've since retrieved the node's = id and used it to "nodetool removenode". After rebalancing, I added the node ba= ck and "nodetool cleaned" up. Everything's up and running, but I'd like to unde= rstand what Cassandra was doing.



------=_Part_1303558_270961676.1483731457367--