Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E06E7200BAF for ; Mon, 17 Oct 2016 07:49:04 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id DF326160AF8; Mon, 17 Oct 2016 05:49:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 06763160AD0 for ; Mon, 17 Oct 2016 07:49:03 +0200 (CEST) Received: (qmail 22323 invoked by uid 500); 17 Oct 2016 05:49:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22313 invoked by uid 99); 17 Oct 2016 05:49:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Oct 2016 05:49:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id B0EBBC16D8 for ; Mon, 17 Oct 2016 05:49:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.779 X-Spam-Level: * X-Spam-Status: No, score=1.779 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=instaclustr-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id hT4tkd1Eew3U for ; Mon, 17 Oct 2016 05:48:59 +0000 (UTC) Received: from mail-qk0-f180.google.com (mail-qk0-f180.google.com [209.85.220.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 810BA5F397 for ; Mon, 17 Oct 2016 05:48:59 +0000 (UTC) Received: by mail-qk0-f180.google.com with SMTP id z190so213671410qkc.2 for ; Sun, 16 Oct 2016 22:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=instaclustr-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=b249kW+Pu+rYW7ctLYaBpJg1Y3Px/J2HlBc1A2pupXs=; b=c83dMeFtq5Nxmzw3aK6MTkidHIW0lKgB+421OC6M79MR6OEyNOwpbNhNzL5A26y/fA qtazS5VeCS+pOypq4/6vMgFuRc3KkHje8LJ0PLPUxaX5SQn0OppLVoaOmH3jbLZwGYMW kUsN/XTLG7zOk2ky6EwVF4j57ifbCACZImgEHc/7Z+XnIHjgQJkle6hD0Inxad//UQP4 CUqIKqt3T89vInP76Cd8tapwDR+b7gveimMiCW3iPcO73vLxogPkXQYtVXo1dgO70+2C U9HJ7SMf2ChB8J/wdSEUWHQTp/hFj0CqMKwerB/xLnDOyCn88fgYZk1nlW2m+ZebCUmZ RE2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=b249kW+Pu+rYW7ctLYaBpJg1Y3Px/J2HlBc1A2pupXs=; b=ZKPTuVWGK8asGEaQeKbMJXvF9TRY0wiHZlS6sDl3lOr/AN0ErKoTWJRyJDz3OBNpWg f6cBB6jiEKqNa/22u2Dai5x1Fj9c7uhb9UqXY1w8JCk8HNFUGx9pJGBdMd8aoC4LRDl8 U3iq74Z8ARPhfsf6E7/5S+IGxlp9r/f/FZ3GfcA/LCVbkImLfY8Rp3mnaT1yjSAXOE7m F5K8EvqCdF2AoG+mBcIcKgDeuAv/XWV/L7i68w/8UvqbESUiLyhXGaXry/91sy4veokJ qHXzxHhxXGNPjGwwZu3NaZ7JU9rw6avJnjPZA+fbx9R2Dn5hLTk30VT6iMA6dM2l5O9D sMUQ== X-Gm-Message-State: AA6/9Rm2q2XEpGe/HszEBvdpv1ejL9PseqIZLbu3hxxkaAPDyMMuCGlbwggatSWzKy3wiVHqf9SpOop+Ca/aDfaS X-Received: by 10.55.147.193 with SMTP id v184mr11626024qkd.2.1476683338335; Sun, 16 Oct 2016 22:48:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ben Slater Date: Mon, 17 Oct 2016 05:48:47 +0000 Message-ID: Subject: Re: failure node rejoin To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=94eb2c08bdfad3d8a3053f092184 archived-at: Mon, 17 Oct 2016 05:49:05 -0000 --94eb2c08bdfad3d8a3053f092184 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable To cassandra, the node where you deleted the files looks like a brand new machine. It doesn=E2=80=99t automatically rebuild machines to prevent accid= ental replacement. You need to tell it to build the =E2=80=9Cnew=E2=80=9D machine= s as a replacement for the =E2=80=9Cold=E2=80=9D machine with that IP by setting -Dcassandra.replace_address_first_boot=3D. See http://cassandra.apache.org/doc/latest/operating/topo_changes.html. Cheers Ben On Mon, 17 Oct 2016 at 16:41 Yuji Ito wrote: > Hi all, > > A failure node can rejoin a cluster. > On the node, all data in /var/lib/cassandra were deleted. > Is it normal? > > I can reproduce it as below. > > cluster: > - C* 2.2.7 > - a cluster has node1, 2, 3 > - node1 is a seed > - replication_factor: 3 > > how to: > 1) stop C* process and delete all data in /var/lib/cassandra on node2 > ($sudo rm -rf /var/lib/cassandra/*) > 2) stop C* process on node1 and node3 > 3) restart C* on node1 > 4) restart C* on node2 > > nodetool status after 4): > Datacenter: datacenter1 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Status=3DUp/Down > |/ State=3DNormal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > DN [node3 IP] ? 256 100.0% > 325553c6-3e05-41f6-a1f7-47436743816f rack1 > UN [node2 IP] 7.76 MB 256 100.0% > 05bdb1d4-c39b-48f1-8248-911d61935925 rack1 > UN [node1 IP] 416.13 MB 256 100.0% > a8ec0a31-cb92-44b0-b156-5bcd4f6f2c7b rack1 > > If I restart C* on node 2 when C* on node1 and node3 are running (without > 2), 3)), a runtime exception happens. > RuntimeException: "A node with address [node2 IP] already exists, > cancelling join..." > > I'm not sure this causes data lost. All data can be read properly just > after this rejoin. > But some rows are lost when I kill&restart C* for destructive tests after > this rejoin. > > Thanks. > > -- =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 Ben Slater Chief Product Officer Instaclustr: Cassandra + Spark - Managed | Consulting | Support +61 437 929 798 --94eb2c08bdfad3d8a3053f092184 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
To cassandra, the node where you deleted the files looks l= ike a brand new machine. It doesn=E2=80=99t automatically rebuild machines = to prevent accidental replacement. You need to tell it to build the =E2=80= =9Cnew=E2=80=9D machines as a replacement for the =E2=80=9Cold=E2=80=9D mac= hine with that IP by setting=C2=A0-Dcassa= ndra.replace_address_first_boot=3D<dead_node_ip>. See=C2=A0http://cassandra.apache.org/doc/latest/operating/topo_changes.html.
Cheers
Ben

On Mon, 17 Oct 2016 at 16:41 Yuji Ito <yuji@imagine-orb.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Hi all,<= div class=3D"gmail_msg">
A failure node can rejoin a cluster.
On t= he node, all data in /var/lib/cassandra were deleted.
Is it normal?

I can reproduce it as below.

clu= ster:
- C* 2.2.7
- a cluster has node1, 2, 3
- node1 is a seed
-=C2=A0replication_factor: 3

how to:
1) stop C* process and delete all data in /var/lib/cassandra on no= de2 ($sudo rm -rf /var/lib/cassandra/*)
2) stop C* process on node1 and node3
3) restart C* on node1
4) restart= C* on node2

nodetool status after 4):
Datacenter: datacenter1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
-- =C2=A0Address =C2=A0 =C2=A0 =C2=A0 =C2=A0Load =C2=A0 =C2=A0 =C2=A0 Tok= ens =C2=A0 =C2=A0 =C2=A0 Owns (effective) =C2=A0Host ID =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Rack
DN =C2=A0[node3 IP] =C2=A0= ? =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0100.0% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0325553= c6-3e05-41f6-a1f7-47436743816f =C2=A0rack1
UN= =C2=A0[node2 IP] =C2=A07.76 MB =C2=A0 =C2=A0 =C2=A0256 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0100.0% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A005bdb1d4-c= 39b-48f1-8248-911d61935925 =C2=A0rack1
UN =C2= =A0[node1 IP] =C2=A0416.13 MB =C2=A0256 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01= 00.0% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0a8ec0a31-cb92-44b0-b156-5bcd= 4f6f2c7b =C2=A0rack1

If I restart C* on node 2 when C* on n= ode1 and node3 are running (without 2), 3)), a runtime exception happens.
RuntimeException: "A node with address [n= ode2 IP] already exists, cancelling join..."

I'm not sur= e this causes data lost. All data can be read properly=C2=A0just after this= rejoin.
But some rows are lost when I kill&a= mp;restart C* for destructive tests after this rejoin.

Thanks.

--
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94
Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798
--94eb2c08bdfad3d8a3053f092184--