From user-return-62240-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Tue Sep 11 12:04:11 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1876418065B for ; Tue, 11 Sep 2018 12:04:09 +0200 (CEST) Received: (qmail 25935 invoked by uid 500); 11 Sep 2018 10:04:08 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 25925 invoked by uid 99); 11 Sep 2018 10:04:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Sep 2018 10:04:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id DE78018F07B for ; Tue, 11 Sep 2018 10:04:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.408 X-Spam-Level: *** X-Spam-Status: No, score=3.408 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLY=1, HTML_FONT_FACE_BAD=0.289, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nNK22JSfgWQF for ; Tue, 11 Sep 2018 10:04:05 +0000 (UTC) Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 4507F5F3AD for ; Tue, 11 Sep 2018 10:04:04 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id w14-v6so11098494plp.6 for ; Tue, 11 Sep 2018 03:04:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=tIBJ4El8NLrsKjFByfwI3nijhurIMalhBV+vMRpQi+A=; b=fAuNCARJfjXFGTJwTr0tDDLEzXp4pcYcxSxK3Tyg6RO/0dUVriwMG/FvaaFSm9SZed KhhR59nDYouyy4Dqt1rV2LB6Ytl73cKTjql7XWW1ByFsD2v2OutdxO5UXACT25/CSbq6 XyCz0GVFFgzxII+GK+Zt494XGok5TC+Re5WgW0TrOj7KOBX6TNTLSxLckNLTR5zbO+ZZ +/ygx185bx9HhyWWzsNZreK/3kX+v7Y99XiE8q05cKs1+Uv9YF+YmjJvIAIF/6PJYwEd zMcR2LwStm+IGp0kp9pBQvABh+HYhsUloMeZOvarSrgZePSz225k0ynWGI881byyuCon 65+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=tIBJ4El8NLrsKjFByfwI3nijhurIMalhBV+vMRpQi+A=; b=V5U9OFj70nzQqSOLnpIEGKJ9hvne0SSCepfe92fn6Es8j49IKjmzv3V40pMs5ac+fb ArMCvZ8oKwRkrf3ulzBLx4vB9tT1C7OP02/U3raxp/wzIFfI9y97639v/ZmImC0Ojsbp XXCYE41wBUYkAgcrj1QQhZ0gbEk6TdMWQUOpv0lG2oZeBPGOdtZ9r2Dqw2l4FrPR2IbH AzW2ltRfn7dvz2EfnmbpxvAj1/EONbQC1VdGlknDa4P1s4Jdc75hmSLjEko465SRoFMa txYUlZuCikdC/6DYn+lrF8IwiytORa+mITV/Va6FQp5vgGKsdJCsnZxxZNXZTuhV/rxd zBeA== X-Gm-Message-State: APzg51ADHuIUX4lmy5U3SiLs0Br9W3jlFNHca6CS8XP6m01ylCjHfhZC bGjlM2EwxJVG5+8To40iuPwQVZXr X-Google-Smtp-Source: ANB0VdY0KW/ifGoSb48Nnw4vUhz+Wm9pP6Di+/aENRLJf97oVIXQ776PNWcbL50K1HAdo0bJ2vbJAw== X-Received: by 2002:a17:902:18a:: with SMTP id b10-v6mr26490715plb.62.1536660241847; Tue, 11 Sep 2018 03:04:01 -0700 (PDT) Received: from [172.26.109.182] ([211.56.96.51]) by smtp.gmail.com with ESMTPSA id f67-v6sm27975136pff.29.2018.09.11.03.04.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 03:04:01 -0700 (PDT) From: Eunsu Kim Content-Type: multipart/alternative; boundary="Apple-Mail=_3183F0AE-4174-40F2-8BDB-659AE7ACCF13" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Default Single DataCenter -> Multi DataCenter Date: Tue, 11 Sep 2018 19:03:57 +0900 References: <65B5EE9A-1D90-4764-BF52-ABFEA127A0D3@gmail.com> <41EF4667-3347-400B-9AAC-EEB7EC5221C0@gmail.com> To: user@cassandra.apache.org In-Reply-To: <41EF4667-3347-400B-9AAC-EEB7EC5221C0@gmail.com> Message-Id: X-Mailer: Apple Mail (2.3445.9.1) --Apple-Mail=_3183F0AE-4174-40F2-8BDB-659AE7ACCF13 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 It=E2=80=99s self respond. Step3 is wrong. Even if it was a SimpleSnitch, changing the dc information will not = start CassandraDaemon with the error log. ERROR [main] 2018-09-11 18:36:30,272 CassandraDaemon.java:708 - Cannot = start node if snitch's data center (pg1) differs from previous data = center (datacenter1). Please fix the snitch configuration, decommission = and rebootstrap this node or use the flag -Dcassandra.ignore_dc=3Dtrue. > On 11 Sep 2018, at 2:25 PM, Eunsu Kim wrote: >=20 > Hello >=20 > Thank you for your responses. >=20 > I=E2=80=99ll share my adding datacenter plan. If you see problems, = please respond. >=20 > The sentence may be a little awkward because I am so poor at English = that I am being helped by a translator. >=20 > I've been most frequently referred = to.(https://medium.com/p/465e9bf28d99 = ) Thank you for your cleanliness. = (Pradeep Chhetri) >=20 > I will also upgrade the version as Alain Rodriguez's advice. >=20 > =E2=80=94=E2=80=94=E2=80=94=E2=80=94 >=20 > Step 1. Stop all existing clusters. (My service is paused.) >=20 > Step 2. Install Cassandra 3.11.3 and copy existing conf files. >=20 > Step 3. Modify cassandra-rackdc.properties for existing nodes. = dc=3Dmydc1 rack=3Dmyrack1 > Q. I think this modification will not affect the existing data = because it was SimpleSnitch before, right? >=20 > Step 4. In the caseandra.yaml of existing nodes, endpoint_snitch is = changed to GossippingPropertyFileSnitch. >=20 > Step 5. Restart the Cassandra of the existing nodes. (My service is = resumed.) >=20 > Step 6. Change the settings of all existing clients to = DCAwareRobinPolicy and refer to mydc1. Consistency level is LOCAL_ONE. = And rolling restart. > Q. Isn't it a problem that at this point, DCAwareRobinPolicy and = RoundRobinPolicy coexist? >=20 > Step 7. Alter my keyspace and system keyspace(system_distributed, = system_traces) : SimpleStrategy(RF=3D2) -> { 'class' : = 'NetworkTopologyStrategy', =E2=80=98mydc1=E2=80=99 : 2 } >=20 > Step 8. Install Cassandra in a new cluster, copying existing conf = files, and setting it to Cassandra-racdc.properties. dc=3Dmydc2 = rack=3Dmyrack2 >=20 > Step 9. Adding a new seed node to the cassandra.yaml of the existing = cluster (mydc1) and restart. > Q1. Must I add the new seed nodes in five all existing nodes? > Q2. Don't I need to update the seed node settings of the new = cluster (mydc2)? >=20 > Step 10. Alter my keyspace and system keyspace(system_distributed, = system_traces) : { 'class' : 'NetworkTopologyStrategy', =E2=80=98mydc1=E2= =80=99 : 1, =E2=80=98mydc2=E2=80=99 : 1 } >=20 > Step 11. Run 'nodetool rebuild =E2=80=94 mydc1=E2=80=99 in turn, in = the new node. >=20 > =E2=80=94=E2=80=94=E2=80=94 >=20 >=20 > I'll run the procedure on the development envrionment and share it. >=20 > Thank you. >=20 >=20 >=20 >=20 >> On 10 Sep 2018, at 10:26 PM, Pradeep Chhetri > wrote: >>=20 >> Hello Eunsu,=20 >>=20 >> I am going through the same exercise at my job. I was making notes as = i was testing the steps in my preproduction environment. Although I = haven't tested end to end but hopefully this might help you: = https://medium.com/p/465e9bf28d99 >>=20 >> Regards, >> Pradeep >>=20 >> On Mon, Sep 10, 2018 at 5:59 PM, Alain RODRIGUEZ > wrote: >> Adding a data center for the first time is a bit tricky when you = haven't been considering it from the start. >>=20 >> I operate 5 nodes cluster (3.11.0) in a single data center with = SimpleSnitch, SimpleStrategy and all client policy RoundRobin. >>=20 >> You will need: >>=20 >> - To change clients, make them 'DCAware'. This depends on the client, = but you should be able to find this in your Cassandra driver (client = side). >> - To change clients, make them use 'LOCAL_' consistency = ('LOCAL_ONE'/'LOCAL_QUORUM' being the most common). >> - To change 'SimpleSnitch' for 'EC2Snitch' or = 'GossipingPropertyFileSnitch' for example, depending on your = context/preference >> - To change 'SimpleStrategy' for 'NetworkTopologyStrategy' for all = the keyspaces, with the desired RF. I take the chance to say that = switching to 1 replica only is often a mistake, you can indeed have data = loss (which you accept) but also service going down, anytime you restart = a node or that a node goes down. If you are ok with RF=3D1, RDBMS might = be a better choice. It's an anti-pattern of some kind to run Cassandra = with RF=3D1. Yet up to you, this is not our topic :). In the same kind = of off-topic recommendations, I would not stick with C*3.11.0, but go to = C*3.11.3 (if you do not perform slice delete, there is still a bug with = this apparently) >>=20 >> So this all needs to be done before starting adding the new data = center. Changing the snitch is tricky, make sure that the new snitch = uses the racks and dc names currently in use in your cluster for the = current cluster, if not the data could not be accessible after the = configuration change. >>=20 >> Then the procedure to add a data center is probably described around. = I know I did this detailed description in 2014, here it is: = https://mail-archives.apache.org/mod_mbox/cassandra-user/201406.mbox/%3CCA= +VSrLopop7Th8nX20aOZ3As75g2jrJm3ryX119dekLYNHqFwA@mail.gmail.com%3E = , = but you might find better/more recent documentation than this one for = this relatively common process, like the documentation you linked. >>=20 >> If you are not confident or have doubts, you can share more about the = context and post your exact plan, as I did years ago in the mail = previously linked. People here should be able to confirm the process is = ok before you move forward, giving you an extra confidence. >>=20 >> C*heers, >> ----------------------- >> Alain Rodriguez - @arodream - alain@thelastpickle.com = >> France / Spain >>=20 >> The Last Pickle - Apache Cassandra Consulting >> http://www.thelastpickle.com >>=20 >> Le lun. 10 sept. 2018 =C3=A0 11:05, Eunsu Kim > a =C3=A9crit : >> Hello everyone >>=20 >> I operate 5 nodes cluster (3.11.0) in a single data center with = SimpleSnitch, SimpleStrategy and all client policy RoundRobin. >>=20 >> At this point, I am going to create clusters of the same size in = different data centers. >>=20 >> I think these two documents are appropriate, but there is confusion = because they are referenced to each other. >>=20 >> = https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCTo= Cluster.html = >> = https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchS= nitch.html = >>=20 >> Anyone who can clearly guide the order? Currently RF is 2 and I want = to have only one replica in the NetworkTopologyStrategy. >> A little data loss is okay. >>=20 >> Thank you in advanced.. >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >=20 --Apple-Mail=_3183F0AE-4174-40F2-8BDB-659AE7ACCF13 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
It=E2=80=99s self = respond.

Step3 = is wrong.

Even if it was a SimpleSnitch, = changing the dc information will not start CassandraDaemon with the = error log.

ERROR [main] 2018-09-11 = 18:36:30,272 CassandraDaemon.java:708 - Cannot start node if snitch's = data center (pg1) differs from previous data center (datacenter1). = Please fix the snitch configuration, decommission and rebootstrap this = node or use the flag -Dcassandra.ignore_dc=3Dtrue.


On 11 Sep 2018, at 2:25 PM, Eunsu Kim <eunsu.bill23@gmail.com> wrote:

Hello

Thank you for your responses.

I=E2=80=99ll share my adding datacenter plan. If = you see problems, please respond.

The = sentence may be a little awkward because I am so poor at English that I = am being helped by a translator.

I've been most frequently referred to.(https://medium.com/p/465e9bf28d99) Thank you for your cleanliness. (Pradeep = Chhetri)

I will also = upgrade the version as Alain Rodriguez's advice.

=E2=80=94=E2=80=94=E2=80=94=E2=80=94

Step 1. Stop all existing clusters. (My service is paused.)

Step 2. Install Cassandra = 3.11.3 and copy existing conf files.

Step 3. Modify = cassandra-rackdc.properties for existing nodes. dc=3Dmydc1 = rack=3Dmyrack1
     Q. I think this = modification will not affect the existing data because it was = SimpleSnitch before, right?

Step 4. In the caseandra.yaml = of existing nodes, endpoint_snitch is changed to = GossippingPropertyFileSnitch.

Step 5. Restart the Cassandra = of the existing nodes. (My service is = resumed.)

Step 6. Change the settings of all existing clients to = DCAwareRobinPolicy and refer to mydc1. Consistency level is LOCAL_ONE. = And rolling restart.
      = Q. Isn't it a problem = that at this point, DCAwareRobinPolicy and RoundRobinPolicy = coexist?

Step 7. Alter my keyspace and system = keyspace(system_distributed, system_traces) :  SimpleStrategy(RF=3D2) = -> { 'class' : 'NetworkTopologyStrategy', =E2=80=98mydc1=E2=80=99 : 2 = }

Step = 8. Install Cassandra in a = new cluster, copying existing conf files, and setting it to = Cassandra-racdc.properties. dc=3Dmydc2 rack=3Dmyrack2

Step = 9. Adding a new seed node = to the cassandra.yaml of the existing cluster (mydc1) and restart.
      Q1. Must I add the new seed nodes in five all = existing nodes?
      = Q2. Don't I need to update = the seed node settings of the new cluster (mydc2)?

Step = 10. Alter my keyspace and system keyspace(system_distributed, = system_traces) :  { 'class' : = 'NetworkTopologyStrategy', =E2=80=98mydc1=E2=80=99 : 1, =E2=80=98mydc= 2=E2=80=99 : 1 }

Step 11. Run 'nodetool rebuild =E2=80=94 mydc1=E2=80=99 in = turn, in the new node.

=E2=80=94=E2=80=94=E2=80=94


I'll run the procedure = on the development envrionment and share it.

Thank = you.




On 10 Sep 2018, at 10:26 PM, Pradeep Chhetri <pradeep@stashaway.com> wrote:

Hello Eunsu, 

I am going through the = same exercise at my job. I was making notes as i was testing the steps = in my preproduction environment. Although I haven't tested end to end = but hopefully this might help you: https://medium.com/p/465e9bf28d99

Regards,
Pradeep

On Mon, Sep 10, 2018 at 5:59 PM, = Alain RODRIGUEZ <arodrime@gmail.com> wrote:
Adding a data center for the first time is a bit = tricky when you haven't been considering it from the start.

I operate 5 nodes cluster = (3.11.0) in a single data center with SimpleSnitch, SimpleStrategy and = all client policy RoundRobin.

You will need:

- To change clients, make them 'DCAware'. This = depends on the client, but you should be able to find this in your = Cassandra driver (client side).
- To change clients, make = them use 'LOCAL_<WHATEVER>' consistency = ('LOCAL_ONE'/'LOCAL_QUORUM' being the most common).
-= To change 'SimpleSnitch' for 'EC2Snitch' or = 'GossipingPropertyFileSnitch' for example, depending on your = context/preference
- To change 'SimpleStrategy' for = 'NetworkTopologyStrategy' for all the keyspaces, with the desired RF. I = take the chance to say that switching to 1 replica only is often a = mistake, you can indeed have data loss (which you accept) but also = service going down, anytime you restart a node or that a node goes down. = If you are ok with RF=3D1, RDBMS might be a better choice. It's an = anti-pattern of some kind to run Cassandra with RF=3D1. Yet up to you, = this is not our topic :). In the same kind of off-topic recommendations, = I would not stick with C*3.11.0, but go to C*3.11.3 (if you do not = perform slice delete, there is still a bug with this = apparently)

So this all needs to be = done before starting adding the new data center. = Changing the snitch is tricky, make sure that the new snitch uses the = racks and dc names currently in use in your cluster for the current = cluster, if not the data could not be accessible after the configuration = change.

Then = the procedure to add a data center is probably described around. I know = I did this detailed description in 2014, here it is: https://mail-archives.apache.org/mod_mbox/cassandra-user/201406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm3ryX119dekLYNHqFwA@mail.gmail.com%3E, but = you might find better/more recent documentation than this one for this = relatively common process, like the documentation you linked.

If you are not confident or have doubts, you = can share more about the context and post your exact plan, as I did = years ago in the mail previously linked. People here should be able to = confirm the process is ok before you move forward, giving you an extra = confidence.

C*heers,
-----------------------
Alain Rodriguez = - @arodream - alain@thelastpickle.com
France / = Spain

The Last = Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
<= /div>

Le lun. 10 sept. = 2018 =C3=A0 11:05, Eunsu Kim <eunsu.bill23@gmail.com> a =C3=A9crit :
Hello everyone

I operate 5 nodes cluster (3.11.0) in a single data center = with SimpleSnitch, SimpleStrategy and all client policy = RoundRobin.

At = this point, I am going to create clusters of the same size in different = data centers.

I = think these two documents are appropriate, but there is confusion = because they are referenced to each other.

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html

Anyone who can clearly = guide the order? Currently RF is 2 and I want to have only one = replica in the NetworkTopologyStrategy.
A little = data loss is okay.

Thank you in advanced..








= --Apple-Mail=_3183F0AE-4174-40F2-8BDB-659AE7ACCF13--