From user-return-63768-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed May 1 10:50:30 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 09963180629 for ; Wed, 1 May 2019 12:50:29 +0200 (CEST) Received: (qmail 73934 invoked by uid 500); 1 May 2019 10:50:26 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 73904 invoked by uid 99); 1 May 2019 10:50:26 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 May 2019 10:50:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D90BD180C96 for ; Wed, 1 May 2019 10:50:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.801 X-Spam-Level: * X-Spam-Status: No, score=1.801 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id t6Xsir8dUMxy for ; Wed, 1 May 2019 10:50:24 +0000 (UTC) Received: from mail-it1-f178.google.com (mail-it1-f178.google.com [209.85.166.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 37E9D5F1F0 for ; Wed, 1 May 2019 10:50:23 +0000 (UTC) Received: by mail-it1-f178.google.com with SMTP id l10so5814751iti.3 for ; Wed, 01 May 2019 03:50:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:date:subject:message-id :references:in-reply-to:to; bh=c0GIVfoSvlsmHf4qb4801Uq+/3urkQgkggesAV3MAW4=; b=rR5TEagiHI87lsLxMEYb38acC8nWOAGxajV4eHp2XekbRCYT07oKK6IAHF2VSo9edf dzH7ZPmkbDlTpvDktw47tXagUa0r1ksV0kmUvnVIboG7Ocu1vc8zvZYj6ehb7C2zvlN6 J9RFl6qGPgW/SNRi3v5pe5jyS46CzoJ5VwbT5DjTWTEadMkYI7ZXi3oNzUw5G/dah2+I SiK0Ed9TpAqKpdTYDaULifKSQbd58dZa7WPcKNvRpB6sIAhis2LDfNrAItfzOJ/fatmJ AZEkpErOtLW69OW1ZGvp/4I+ZdadtOndrNlPi7aUiQt1fjkXZtcDUZFNvpqhp7IwsHS7 jWKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:references:in-reply-to:to; bh=c0GIVfoSvlsmHf4qb4801Uq+/3urkQgkggesAV3MAW4=; b=V1yJGa/kku1vvVPPhrYIwvPY39I+lPTjYeKmtMh5ySmvFSKHtaqh49bbCanF+2L6Ym 8y+FJgEIgPndk5Ia0UNcIjbbIqGswk74lMcByqofwCiRSQP2JmhhKwTzz6LaFtWuBH/e 91adyrRfQNzZ00dMrhP+osalk0OSxn2nFwUjbRI0L8Ju7RfWTWi5iaxTUpXvH5LYXrWt 06UF0sAV0BD067ReXcVvg/aHDTJMzqm0U/D0uk/tD07GpQ+FTY9LXJE4UJvRHleuUyph RWbCGcC03tpoaiAztvN4/ocDUjl2NNy0s760+9pLNuMFJFMNQkgyKBkSU8vj77gvwOAl 2Fmw== X-Gm-Message-State: APjAAAXlwa/DdAraHJNIihhFW2OVIfc05k5Nr8Z4XQ04aiYSX7lW1GVc A9qookoj5xT81baHqTHYhYhNQKoi X-Google-Smtp-Source: APXvYqwbHkp33Gi9b7uiLNnr/EVWKfXgx4kvPQw44LhEZSgVTNp15cYufHWmWHIcn3tBTBfNe3d1lA== X-Received: by 2002:a24:4584:: with SMTP id c4mr7585255itd.163.1556707821784; Wed, 01 May 2019 03:50:21 -0700 (PDT) Received: from ?IPv6:2601:402:4500:384d:58fd:e5b8:b9cb:f086? ([2601:402:4500:384d:58fd:e5b8:b9cb:f086]) by smtp.gmail.com with ESMTPSA id w12sm12454891ioc.4.2019.05.01.03.50.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 May 2019 03:50:21 -0700 (PDT) From: Fred Habash Content-Type: multipart/alternative; boundary=Apple-Mail-C0AE690C-FB75-47B9-9398-625EB6ABE416 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Date: Wed, 1 May 2019 06:50:20 -0400 Subject: Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees Message-Id: References: <5cc8ac26.1c69fb81.81186.b30f@mx.google.com> In-Reply-To: To: user@cassandra.apache.org X-Mailer: iPhone Mail (16E227) --Apple-Mail-C0AE690C-FB75-47B9-9398-625EB6ABE416 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Thank you.=20 Range movement is one reason this is enforced when adding a new node. But, w= hat about forcing a consistent bootstrap i.e. bootstrapping from primary own= er of the range and not a secondary replica.=20 How=E2=80=99s consistent bootstrap enforced when replacing a dead node.=20 =E2=80=94=E2=80=94=E2=80=94=E2=80=94- Thank you.=20 > On Apr 30, 2019, at 7:40 PM, Alok Dwivedi w= rote: >=20 > When a new node joins the ring, it needs to own new token ranges. This sho= uld be unique to the new node and we don=E2=80=99t want to end up in a situa= tion where two nodes joining simultaneously can own same range (and ideally e= venly distributed). Cassandra has this 2 minute wait rule for gossip state t= o propagate before a node is added. But this on its does not guarantees tha= t token ranges can=E2=80=99t overlap. See this ticket for more details https= ://issues.apache.org/jira/browse/CASSANDRA-7069 To overcome this issue, the= approach was to only allow one node joining at a time.=20 > =20 > When you replace a dead node the new token range selection does not applie= s as the replacing node just owns the token ranges of the dead node. I think= that=E2=80=99s why the restriction of only replacing one node at a time doe= s not applies in this case.=20 > =20 > =20 > Thanks > Alok Dwivedi > Senior Consultant > https://www.instaclustr.com/platform/ > =20 > =20 > =20 > =20 > =20 > From: Fd Habash > Reply-To: "user@cassandra.apache.org" > Date: Wednesday, 1 May 2019 at 06:18 > To: "user@cassandra.apache.org" > Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consi= stency Guarantees > =20 > Reviewing the documentation & based on my testing, using C* 2.2.8, I was n= ot able to extend the cluster by adding multiple nodes simultaneously. I got= an error message =E2=80=A6 > =20 > Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while c= assandra.consistent.rangemovement is true > =20 > I understand this is to force a node to bootstrap from the former owner of= the range when adding a node as part of extending the cluster. > =20 > However, I was able to bootstrap multiple nodes to replace dead nodes. C* d= id not complain about it. > =20 > Is consistent range movement & the guarantee it offers to bootstrap from p= rimary range owner not applicable when bootstrapping to replace dead nodes? > =20 > ---------------- > Thank you > =20 --Apple-Mail-C0AE690C-FB75-47B9-9398-625EB6ABE416 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Thank you. 

Range movemen= t is one reason this is enforced when adding a new node. But, what about for= cing a consistent bootstrap i.e. bootstrapping from primary owner of the ran= ge and not a secondary replica. 

How=E2=80=99s= consistent bootstrap enforced when replacing a dead node. 


=E2=80=94=E2=80=94=E2=80=94=E2= =80=94-
Thank you. 

On Apr 30= , 2019, at 7:40 PM, Alok Dwivedi <alok.dwivedi@instaclustr.com> wrote:

When a new node joins the ring, it needs to own new t= oken ranges. This should be unique to the new node and we don=E2=80=99t want= to end up in a situation where two nodes joining simultaneously can own sam= e range (and ideally evenly distributed). Cassandra has this 2 minute wait rule for gossip state to propagate before a= node is added.  But this on its does not guarantees that token ranges c= an=E2=80=99t overlap. See this ticket for more details https://iss= ues.apache.org/jira/browse/CASSANDRA-7069 To overcome this  issue, t= he approach was to only allow one node joining at a time. 

 

When you replace a dead node the new token range sele= ction does not applies as the replacing node just owns the token ranges of t= he dead node. I think that=E2=80=99s why the restriction of only replacing o= ne node at a time does not applies in this case. 

 

 

Thanks

Alok Dwivedi

Senior Consultant

https://www.instaclustr.com/platform/<= o:p>

 

 

 

 

 

From:= Fd Habash <fmhabash@gmail.com>
Reply-To: "user@cassandr= a.apache.org" <user@cass= andra.apache.org>
Date: Wednesday, 1 May 2019 at 06:18
To: "user@cassandra.apac= he.org" <user@cassandra.= apache.org>
Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: C= onsistency Guarantees

 

Reviewing the documentation &  based on my t= esting, using C* 2.2.8, I was not able to extend the cluster by adding multi= ple nodes simultaneously. I got an error message =E2=80=A6

 

Other bootstrapping/leaving/moving nodes detected, ca= nnot bootstrap while cassandra.consistent.rangemovement is true

 

I understand this is to force a node to bootstrap fro= m the former owner of the range when adding a node as part of extending the c= luster.

 

However, I was able to bootstrap multiple nodes to re= place dead nodes. C* did not complain about it.

 

Is consistent range movement & the guarantee it o= ffers to bootstrap from primary range owner not applicable when bootstrappin= g to replace dead nodes?

 

----------------
Thank you

 

= --Apple-Mail-C0AE690C-FB75-47B9-9398-625EB6ABE416--