Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4BD22200BB6 for ; Fri, 21 Oct 2016 08:00:19 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 48B3F160AF2; Fri, 21 Oct 2016 06:00:19 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 65315160AE0 for ; Fri, 21 Oct 2016 08:00:16 +0200 (CEST) Received: (qmail 6619 invoked by uid 500); 21 Oct 2016 06:00:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 6608 invoked by uid 99); 21 Oct 2016 06:00:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2016 06:00:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9B0061A0146 for ; Fri, 21 Oct 2016 06:00:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.779 X-Spam-Level: * X-Spam-Status: No, score=1.779 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=imagine-orb-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id B2hiMqY9DNI4 for ; Fri, 21 Oct 2016 06:00:09 +0000 (UTC) Received: from mail-qk0-f180.google.com (mail-qk0-f180.google.com [209.85.220.180]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 36BB15FAE0 for ; Fri, 21 Oct 2016 06:00:08 +0000 (UTC) Received: by mail-qk0-f180.google.com with SMTP id f128so126440042qkb.1 for ; Thu, 20 Oct 2016 23:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=imagine-orb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Doz6BHFuqh2H/I3XTqIfdY1fjguLyuePtF3P6meKUeQ=; b=LiIt/L8mNQsSJkL2IIkRhfCOBOoPq7ipzSFEac5pJewovMiOA7suaO4zSwFu/qNCpE j5IC3YZ+j84lCMfsDrO4QZJF1nWNUsLk4tz8UQPaMgafmHfUe07wHN/JPlFBsRl0GHig 1+pzhiEPqXEuL4UPyyq3YX4GgE9RQEWBL//nT51a2XCBUhtcIe7mJCa4UOPgLwQ3DT6K RG7Th96fTptbpBDIEkPLJWV9lq/2RiodHqk4L3YF49U1qx3+HDF7repFgK19BMS65s5G N4c/HtFRe6eszbGVKjiVnJYza6XsL+O7Z6U7CvIYmTCVNjaXV5iSbaFtKmL9XEyCyWtq 1ERw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Doz6BHFuqh2H/I3XTqIfdY1fjguLyuePtF3P6meKUeQ=; b=ZDjI291LneKb5eqEb3oV1j2rzv3ZnmexndpuV7FY6izs4lVTvaG2MRL2nXJV57efbj tWCy8x8T0XsDWgzR58UabfaY2a0JE++PPwg8yDxSN+nkugAeNG+zfhovAwCsW/QaAc5q pYnPwR/OauCHfe3emKhP7HP8sQ3iYHm17bU6zOTMx5+Tp1OKshOfP7I7JMKk9H0byQdi AAmOKwE3M7p++/z2xRe72D9wLYmZqgzEhONyDsJCt4f89jmC47BWY+2ab+hHj/SbtEh+ fN5j0DmwSSlHZMoeNV2NTZHTawrh+wR4EOtJqD3YrsP3pNdN/sC7sQQnl/0JqF/Ssefe l5CQ== X-Gm-Message-State: ABUngvcA70P3Zh77BgAuu/KQWPk1eGuYyCywj32DtxigaoxGFMRqIRgYkQZYgcQh087r1da4mYXHGOx0yLBCsA== X-Received: by 10.194.75.165 with SMTP id d5mr2472903wjw.190.1477029605323; Thu, 20 Oct 2016 23:00:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.109.149 with HTTP; Thu, 20 Oct 2016 23:00:04 -0700 (PDT) In-Reply-To: References: <5C39A00F-4569-41E4-80F7-07651E66CAB5@crowdstrike.com> From: Yuji Ito Date: Fri, 21 Oct 2016 15:00:04 +0900 Message-ID: Subject: Re: failure node rejoin To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7bb04bc2f2d28a053f59c011 archived-at: Fri, 21 Oct 2016 06:00:19 -0000 --047d7bb04bc2f2d28a053f59c011 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > Are you certain your tests don=E2=80=99t generate any overlapping inserts= (by PK)? Yes. The operation 2) also checks the number of rows just after all insertions. On Fri, Oct 21, 2016 at 2:51 PM, Ben Slater wrote: > OK. Are you certain your tests don=E2=80=99t generate any overlapping ins= erts (by > PK)? Cassandra basically treats any inserts with the same primary key as > updates (so 1000 insert operations may not necessarily result in 1000 row= s > in the DB). > > On Fri, 21 Oct 2016 at 16:30 Yuji Ito wrote: > >> thanks Ben, >> >> > 1) At what stage did you have (or expect to have) 1000 rows (and have >> the mismatch between actual and expected) - at that end of operation (2)= or >> after operation (3)? >> >> after operation 3), at operation 4) which reads all rows by cqlsh with >> CL.SERIAL >> >> > 2) What replication factor and replication strategy is used by the tes= t >> keyspace? What consistency level is used by your operations? >> >> - create keyspace testkeyspace WITH REPLICATION =3D >> {'class':'SimpleStrategy','replication_factor':3}; >> - consistency level is SERIAL >> >> >> On Fri, Oct 21, 2016 at 12:04 PM, Ben Slater >> wrote: >> >> >> A couple of questions: >> 1) At what stage did you have (or expect to have) 1000 rows (and have th= e >> mismatch between actual and expected) - at that end of operation (2) or >> after operation (3)? >> 2) What replication factor and replication strategy is used by the test >> keyspace? What consistency level is used by your operations? >> >> >> Cheers >> Ben >> >> On Fri, 21 Oct 2016 at 13:57 Yuji Ito wrote: >> >> Thanks Ben, >> >> I tried to run a rebuild and repair after the failure node rejoined the >> cluster as a "new" node with -Dcassandra.replace_address_first_boot. >> The failure node could rejoined and I could read all rows successfully. >> (Sometimes a repair failed because the node cannot access other node. If >> it failed, I retried a repair) >> >> But some rows were lost after my destructive test repeated (after about >> 5-6 hours). >> After the test inserted 1000 rows, there were only 953 rows at the end o= f >> the test. >> >> My destructive test: >> - each C* node is killed & restarted at the random interval (within abou= t >> 5 min) throughout this test >> 1) truncate all tables >> 2) insert initial rows (check if all rows are inserted successfully) >> 3) request a lot of read/write to random rows for about 30min >> 4) check all rows >> If operation 1), 2) or 4) fail due to C* failure, the test retry the >> operation. >> >> Does anyone have the similar problem? >> What causes data lost? >> Does the test need any operation when C* node is restarted? (Currently, = I >> just restarted C* process) >> >> Regards, >> >> >> On Tue, Oct 18, 2016 at 2:18 PM, Ben Slater >> wrote: >> >> OK, that=E2=80=99s a bit more unexpected (to me at least) but I think th= e >> solution of running a rebuild or repair still applies. >> >> On Tue, 18 Oct 2016 at 15:45 Yuji Ito wrote: >> >> Thanks Ben, Jeff >> >> Sorry that my explanation confused you. >> >> Only node1 is the seed node. >> Node2 whose C* data is deleted is NOT a seed. >> >> I restarted the failure node(node2) after restarting the seed node(node1= ). >> The restarting node2 succeeded without the exception. >> (I couldn't restart node2 before restarting node1 as expected.) >> >> Regards, >> >> >> On Tue, Oct 18, 2016 at 1:06 PM, Jeff Jirsa >> wrote: >> >> The unstated "problem" here is that node1 is a seed, which implies >> auto_bootstrap=3Dfalse (can't bootstrap a seed, so it was almost certain= ly >> setup to start without bootstrapping). >> >> That means once the data dir is wiped, it's going to start again without >> a bootstrap, and make a single node cluster or join an existing cluster = if >> the seed list is valid >> >> >> >> -- >> Jeff Jirsa >> >> >> On Oct 17, 2016, at 8:51 PM, Ben Slater >> wrote: >> >> OK, sorry - I think understand what you are asking now. >> >> However, I=E2=80=99m still a little confused by your description. I thin= k your >> scenario is: >> 1) Stop C* on all nodes in a cluster (Nodes A,B,C) >> 2) Delete all data from Node A >> 3) Restart Node A >> 4) Restart Node B,C >> >> Is this correct? >> >> If so, this isn=E2=80=99t a scenario I=E2=80=99ve tested/seen but I=E2= =80=99m not surprised Node >> A starts succesfully as there are no running nodes to tell it via gossip >> that it shouldn=E2=80=99t start up without the =E2=80=9Creplaces=E2=80= =9D flag. >> >> I think that right way to recover in this scenario is to run a nodetool >> rebuild on Node A after the other two nodes are running. You could >> theoretically also run a repair (which would be good practice after a we= ird >> failure scenario like this) but rebuild will probably be quicker given y= ou >> know all the data needs to be re-streamed. >> >> Cheers >> Ben >> >> On Tue, 18 Oct 2016 at 14:03 Yuji Ito wrote: >> >> Thank you Ben, Yabin >> >> I understood the rejoin was illegal. >> I expected this rejoin would fail with the exception. >> But I could add the failure node to the cluster without the >> exception after 2) and 3). >> I want to know why the rejoin succeeds. Should the exception happen? >> >> Regards, >> >> >> On Tue, Oct 18, 2016 at 1:51 AM, Yabin Meng wrote: >> >> The exception you run into is expected behavior. This is because as Ben >> pointed out, when you delete everything (including system schemas), C* >> cluster thinks you're bootstrapping a new node. However, node2's IP is >> still in gossip and this is why you see the exception. >> >> I'm not clear the reasoning why you need to delete C* data directory. >> That is a dangerous action, especially considering that you delete syste= m >> schemas. If in any case the failure node is gone for a while, what you n= eed >> to do is to is remove the node first before doing "rejoin". >> >> Cheers, >> >> Yabin >> >> On Mon, Oct 17, 2016 at 1:48 AM, Ben Slater >> wrote: >> >> To cassandra, the node where you deleted the files looks like a brand ne= w >> machine. It doesn=E2=80=99t automatically rebuild machines to prevent ac= cidental >> replacement. You need to tell it to build the =E2=80=9Cnew=E2=80=9D mach= ines as a >> replacement for the =E2=80=9Cold=E2=80=9D machine with that IP by settin= g >> -Dcassandra.replace_address_first_boot=3D. See >> http://cassandra.apache.org/doc/latest/operating/topo_changes.html >> >> . >> >> Cheers >> Ben >> >> On Mon, 17 Oct 2016 at 16:41 Yuji Ito wrote: >> >> Hi all, >> >> A failure node can rejoin a cluster. >> On the node, all data in /var/lib/cassandra were deleted. >> Is it normal? >> >> I can reproduce it as below. >> >> cluster: >> - C* 2.2.7 >> - a cluster has node1, 2, 3 >> - node1 is a seed >> - replication_factor: 3 >> >> how to: >> 1) stop C* process and delete all data in /var/lib/cassandra on node2 >> ($sudo rm -rf /var/lib/cassandra/*) >> 2) stop C* process on node1 and node3 >> 3) restart C* on node1 >> 4) restart C* on node2 >> >> nodetool status after 4): >> Datacenter: datacenter1 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> Status=3DUp/Down >> |/ State=3DNormal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host ID >> Rack >> DN [node3 IP] ? 256 100.0% >> 325553c6-3e05-41f6-a1f7-47436743816f rack1 >> UN [node2 IP] 7.76 MB 256 100.0% >> 05bdb1d4-c39b-48f1-8248-911d61935925 rack1 >> UN [node1 IP] 416.13 MB 256 100.0% >> a8ec0a31-cb92-44b0-b156-5bcd4f6f2c7b rack1 >> >> If I restart C* on node 2 when C* on node1 and node3 are running (withou= t >> 2), 3)), a runtime exception happens. >> RuntimeException: "A node with address [node2 IP] already exists, >> cancelling join..." >> >> I'm not sure this causes data lost. All data can be read properly just >> after this rejoin. >> But some rows are lost when I kill&restart C* for destructive tests afte= r >> this rejoin. >> >> Thanks. >> >> -- >> =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 >> Ben Slater >> Chief Product Officer >> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >> +61 437 929 798 >> >> >> >> -- >> =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 >> Ben Slater >> Chief Product Officer >> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >> +61 437 929 798 >> >> ____________________________________________________________________ >> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential >> and may be legally privileged. If you are not the intended recipient, do >> not disclose, copy, distribute, or use this email or any attachments. If >> you have received this in error please let the sender know and then dele= te >> the email and all attachments. >> >> >> -- >> =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 >> Ben Slater >> Chief Product Officer >> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >> +61 437 929 798 >> >> >> -- >> =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 >> Ben Slater >> Chief Product Officer >> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >> +61 437 929 798 >> >> >> -- > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > Ben Slater > Chief Product Officer > Instaclustr: Cassandra + Spark - Managed | Consulting | Support > +61 437 929 798 > --047d7bb04bc2f2d28a053f59c011 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
>=C2=A0Are you certain= your tests don=E2=80=99t generate any overlapping inserts (by PK)?<= div>
Yes. The operation 2) also checks the number of rows just after = all insertions.

On Fri, Oct 21, 2016 at 2:51 PM, Ben Slater <ben.slater@instac= lustr.com> wrote:
OK. Are you certain your tests don=E2=80=99t generate any overlapp= ing inserts (by PK)? Cassandra basically treats any inserts with the same p= rimary key as updates (so 1000 insert operations may not necessarily result= in 1000 rows in the DB).

=
On Fri, 21 Oct 2016 at 16:30 Yu= ji Ito <yuji@i= magine-orb.com> wrote:
thanks Ben,
<= div dir=3D"ltr" class=3D"m_8997792290516328679gmail_msg">

> 1) At what stage did you have (or expect = to have) 1000 rows (and have the mismatch between actual and expected) - at= that end of operation (2) or after operation (3)?

after operation 3), at operation 4) which reads all rows b= y cqlsh with CL.SERIAL

> 2) Wha= t replication factor and replication strategy is used by the test keyspace?= What consistency level is used by your operations?

- create = keyspace testkeyspace WITH REPLICATION =3D {'class':'SimpleStra= tegy','replication_factor':3};
- consistency level is SERIAL

=

On Fri, Oct 21, 2016 at 12:04 PM, Ben Slater <ben.slater@instaclustr.com> wrote:
=

A coupl= e of questions:
1) At what stage did you have (or expe= ct to have) 1000 rows (and have the mismatch between actual and expected) -= at that end of operation (2) or after operation (3)?
2) What replication factor and replication st= rategy is used by the test keyspace? What consistency level is used by your= operations?


Cheers
Ben

On Fri, 21 Oct 2016 at 13:57 Yuji Ito <yuji@imagine-orb.com> wrote:
Th= anks Ben,

I tr= ied to run a rebuild and repair after the failure node rejoined the cluster= as a "new" node with a= ddress_first_boot.
The failure node could rejo= ined and I could read all rows successfully.
(Sometimes a repair failed because the node cannot acces= s other node. If it failed, I retried a repair)

But some rows were lost after my de= structive test repeated (after about 5-6 hours).
After the test inserted 1000 rows, there were only 9= 53 rows at the end of the test.
<= /div>

My destr= uctive test:
- each C* nod= e is killed & restarted at the random interval (within about 5 min)=C2= =A0throughout this test
1) truncate all tables
2) insert initial rows (check if all row= s are inserted successfully)
3) request a lot of read/write to random rows for about 30min
4) check all rows
If operation 1), 2) or 4) fail due to C* fai= lure, the test retry the operation.

Does anyone have the similar problem?
What causes data lost?
Does the test need any operation when C* node is restarted? (Currentl= y, I just restarted C* process)

Regards,


On Tue, = Oct 18, 2016 at 2:18 PM, Ben Slater <ben.slater@instaclustr.= com> wrote:
OK, that=E2=80=99s a bit more unexpected = (to me at least) but I think the solution of running a rebuild or repair st= ill applies.

On Tue, 18 = Oct 2016 at 15:45 Yuji Ito <yuji@imagine-orb.com> wrote:

Sorry that my expla= nation confused you.

Only node1 is the= seed node.
Node2 whose C* data is deleted is NOT a seed.

I restarted the failure node(node2) after= restarting the seed node(node1).
The restarting node2 succee= ded without the exception.
(I couldn't restart node2 befo= re restarting node1 as expected.)

Regards,


On Tue, Oct 18, 2016 at 1:06 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
The unstated "problem= " here is that node1 is a seed, which implies auto_bootstrap=3Dfalse (= can't bootstrap a seed, so it was almost certainly setup to start witho= ut bootstrapping).

That means once the data dir is wiped, it's = going to start again without a bootstrap, and make a single node cluster or= join an existing cluster if the seed list is valid



--=C2=A0
Jeff Jirsa

<= div class=3D"m_8997792290516328679m_-29665851870655295m_-123788290118807940= 0m_6004363168911021151m_7140314446020094232m_2765514738712472439h5 m_899779= 2290516328679m_-29665851870655295m_-1237882901188079400m_600436316891102115= 1m_7140314446020094232gmail_msg m_8997792290516328679m_-29665851870655295m_= -1237882901188079400gmail_msg m_8997792290516328679gmail_msg">

On Oct 17, 2016, at 8:51 PM, Ben Slater <ben.slater@instaclustr.com>= wrote:

OK, sorry - I think understand what you are asking now.=C2=A0=

However, I=E2=80=99m still a little confuse= d by your description. I think your scenario is:
1) Stop C* o= n all nodes in a cluster (Nodes A,B,C)
2) Delete all data fro= m Node A
3) Restart Node A
4) Restart Node B,C<= /div>

Is this correct?

If so, this isn=E2=80=99t a scenario I=E2=80=99ve tested/se= en but I=E2=80=99m not surprised Node A starts succesfully as there are no = running nodes to tell it via gossip that it shouldn=E2=80=99t start up with= out the =E2=80=9Creplaces=E2=80=9D flag.

I think that right way to recover in this scenario is to run a nodetool = rebuild on Node A after the other two nodes are running. You could theoreti= cally also run a repair (which would be good practice after a weird failure= scenario like this) but rebuild will probably be quicker given you know al= l the data needs to be re-streamed.

C= heers
Ben

On Tue, 18 Oct 2016 at 14:03 Yuji Ito <yuji@imagine-orb.com> wrote:<= br class=3D"m_8997792290516328679m_-29665851870655295m_-1237882901188079400= m_6004363168911021151m_7140314446020094232gmail_msg m_8997792290516328679m_= -29665851870655295m_-1237882901188079400gmail_msg m_8997792290516328679gmai= l_msg">
Thank you= Ben, Yabin

I understoo= d the rejoin was illegal.
I expected this rejoin would= fail with the exception.
But I could a= dd the failure node to the cluster without the exception=C2=A0after 2) and = 3).
I want to know why the rejoin succeeds. Should the= exception happen?



On Tue, Oct 18, 2016= at 1:51 AM, Yabin Meng <yabinmeng@gmail.com&g= t; wrote:
The exception you run into is expected behavior. This is be= cause as Ben pointed out, when you delete everything (including system sche= mas), C* cluster thinks you're bootstrapping a new node. However, =C2= =A0node2's IP is still in gossip and this is why you see the exception.=

I'm not clear the = reasoning why you need to delete C* data directory. That is a dangerous act= ion, especially considering that you delete system schemas. If in any case = the failure node is gone for a while, what you need to do is to is remove t= he node first before doing "rejoin".

Cheers,

<= /div>
Yabin

On Mon, Oct 17, 2016 at 1:48 AM, Ben= Slater <ben.slater@instaclustr.com>= wrote:
To cassandra, the node where you deleted the files looks like= a brand new machine. It doesn=E2=80=99t automatically rebuild machines to = prevent accidental replacement. You need to tell it to build the =E2=80=9Cn= ew=E2=80=9D machines as a replacement for the =E2=80=9Cold=E2=80=9D machine= with that IP by setting=C2=A0= -Dcassandra.replace_address_first_boot=3D<dead_node_ip>. Se= e=C2=A0http://cassandra.apache.org/doc/latest/ope= rating/topo_changes.html.

Cheers
Ben

On Mon, 17 Oct 2016 at 16:41 Yuji Ito <yuji@imagine-orb= .com> wrote:
Hi all,

A = failure node can rejoin a cluster.
On the node, all data = in /var/lib/cassandra were deleted.
Is it normal?

I can reproduce it as below.

cluster:
- C= * 2.2.7
- a cluster has node1, 2, 3
- node1 is a seed
-=C2=A0replicatio= n_factor: 3

how to:
<= div class=3D"m_8997792290516328679m_-29665851870655295m_-123788290118807940= 0m_6004363168911021151m_7140314446020094232m_2765514738712472439m_418526089= 2404546303m_-1588653945541502463m_9044310476665316456m_-2041723423928547865= gmail_msg m_8997792290516328679m_-29665851870655295m_-1237882901188079400m_= 6004363168911021151m_7140314446020094232m_2765514738712472439m_418526089240= 4546303gmail_msg m_8997792290516328679m_-29665851870655295m_-12378829011880= 79400m_6004363168911021151m_7140314446020094232gmail_msg m_8997792290516328= 679m_-29665851870655295m_-1237882901188079400gmail_msg m_899779229051632867= 9gmail_msg">1) stop C* process and delete all data in /var/lib/cassandra on= node2 ($sudo rm -rf /var/lib/cassandra/*)
2)= stop C* process on node1 and node3
3) restart C* on node= 1
4) restart C* on node2

nodetool status after 4):
D= atacenter: datacenter1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
-- =C2=A0Address =C2=A0 =C2=A0 =C2=A0 =C2=A0Load =C2=A0 =C2=A0 =C2=A0 T= okens =C2=A0 =C2=A0 =C2=A0 Owns (effective) =C2=A0Host ID =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Rack
DN =C2=A0[node3 IP] =C2=A0? =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0100.0% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0325553c6-3e05-41f= 6-a1f7-47436743816f =C2=A0rack1
UN =C2=A0[node2 IP] = =C2=A07.76 MB =C2=A0 =C2=A0 =C2=A0256 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0100= .0% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A005bdb1d4-c39b-48f1-8248-9= 11d61935925 =C2=A0rack1
UN =C2=A0[node1 IP] =C2=A0416.13 = MB =C2=A0256 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0100.0% =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0a8ec0a31-cb92-44b0-b156-5bcd4f6f2c7b =C2=A0rack1

If I restart C* on no= de 2 when C* on node1 and node3 are running (without 2), 3)), a runtime exc= eption happens.
RuntimeException: "A node with addre= ss [node2 IP] already exists, cancelling join..."
I'm not sure this causes data lost. All da= ta can be read properly=C2=A0just after this rejoin.
But = some rows are lost when I kill&restart C* for destructive tests after t= his rejoin.

Thanks.
<= div class=3D"m_8997792290516328679m_-29665851870655295m_-123788290118807940= 0m_6004363168911021151m_7140314446020094232m_2765514738712472439m_418526089= 2404546303m_-1588653945541502463m_9044310476665316456m_-2041723423928547865= gmail_msg m_8997792290516328679m_-29665851870655295m_-1237882901188079400m_= 6004363168911021151m_7140314446020094232m_2765514738712472439m_418526089240= 4546303gmail_msg m_8997792290516328679m_-29665851870655295m_-12378829011880= 79400m_6004363168911021151m_7140314446020094232gmail_msg m_8997792290516328= 679m_-29665851870655295m_-1237882901188079400gmail_msg m_899779229051632867= 9gmail_msg">
--
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94
Ben Slater
= Chief Product Officer
Instaclustr: Cassandra + Spark -= Managed | Consulting | Support
+61 4= 37 929 798


--
=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94
Ben Slater
C= hief Product Officer
Instaclustr: Cassandra + Spark - Managed= | Consulting | Support
+61 437 929 798
_________________________________________________________________= ___
CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and = may be legally privileged. If you are not the intended recipient, do not di= sclose, copy, distribute, or use this email or any attachments. If you have= received this in error please let the sender know and then delete the emai= l and all attachments.

= --
=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94
Ben Slater
Chief Product Of= ficer
Instaclustr: Cassand= ra + Spark - Managed | Consulting | Support
+61 437 929 = 798

<= /div>
--
=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94
Ben Slater
Chief = Product Officer
Instaclu= str: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

--
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94
Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support

--047d7bb04bc2f2d28a053f59c011--