Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 577DA10FF7 for ; Thu, 21 Nov 2013 15:44:10 +0000 (UTC) Received: (qmail 31511 invoked by uid 500); 21 Nov 2013 15:44:05 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 31465 invoked by uid 500); 21 Nov 2013 15:44:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 31453 invoked by uid 99); 21 Nov 2013 15:44:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Nov 2013 15:44:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.214.169] (HELO mail-ob0-f169.google.com) (209.85.214.169) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Nov 2013 15:43:56 +0000 Received: by mail-ob0-f169.google.com with SMTP id wm4so6481013obc.14 for ; Thu, 21 Nov 2013 07:43:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=spIvfH0lfTEMPq8rBgpjM0bQ3Uwju1iBCv33udY7gJA=; b=dQuMk8K51Y5vd22YfLaDIab8ITOuMZsLgfUOKYDMwRa2ya/uajzpwp/byey0hvN9Pp vFybTHwTwZDW3LXqjajHcO1tU52MJG5a5NJGMf8JgMtXNpMBUT1FzAihlWGn6RJlBqkf Lm7lNEutGEyB0k3rUMDdsPYKLtjre7GBBMtsIl7XpCAkU5GVtBTHKopejUr4MbwbRlsN 5iOiiDAjydzwQTwrqqkWsMFFscgff+WfCVXyk/92IEZqVZjYw7eDE4Fek0WaRGUVQedL 9QoVbslaAUf6lN2M9NyiV8Z3wMhyyC4YL8YLPoHsK9kKyKoQlt5QGoj76lL7UHr3ixJp onUA== X-Gm-Message-State: ALoCoQkl3JwKv6/e7TrtI17e1IcJO5qPyAnjbCd9RwvdxXlu0ERbdhxkC8Z0IFOPrA8bo8sQ4YwR MIME-Version: 1.0 X-Received: by 10.60.99.71 with SMTP id eo7mr1441014oeb.61.1385048615480; Thu, 21 Nov 2013 07:43:35 -0800 (PST) Received: by 10.76.92.73 with HTTP; Thu, 21 Nov 2013 07:43:35 -0800 (PST) X-Originating-IP: [62.90.66.249] In-Reply-To: References: Date: Thu, 21 Nov 2013 17:43:35 +0200 Message-ID: Subject: Re: Simple test of adding a node causes data loss From: Tamar Rosen To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b33d414b901e604ebb1c20c X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33d414b901e604ebb1c20c Content-Type: text/plain; charset=ISO-8859-1 This worked, thanks. On Thu, Nov 21, 2013 at 5:20 PM, Julien Campan wrote: > Hi, > > You said : Adjusted cassandra.yaml as above except that for seeds put both > server addresses > > If you put the new node into the seeds list, your node will not bootstrap. > This could explain why you see only the half of your data. > > Can you check in system.log ? > > > By the way, you can retry without adding the new server address in the > seeds list and normally should work :) > > You should add your new node into the seeds list only after the bootstrap > operation. > > > > Julien Campan > > > > > 2013/11/21 Tamar Rosen > >> Hi, >> >> We are testing the process of adding a node to a cluster using a simple >> procedure, and seeing data loss. >> >> System: Ubuntu 12.04 on AWS >> Version: Cassandra + dsc 1.2.10 >> >> Here is what we did: >> Created 2 new m1.large instances >> Installed Java >> Installed Cassandra 1.2.10 (the version we are using in our production >> system) >> >> In server1: >> Adjusted cassandra.yaml >> comment out the initial_token >> uncomment num_tokens: 256 >> changed "seeds" to the address of this server >> changed listen_address to the address of this server >> changed rpc_address to 0.0.0.0 >> changed practitioner to org.apache.cassandra.dht.RandomPartitioner >> made sure cassandra is not running >> sudo rm -rf /var/lib/cassandra/* >> started cassandra >> connected via cqlsh >> Created a new keyspace with replication factor 1 >> Created a new table >> Populated the table with 4000 row of simple data using cql copy command >> cqlsh> select count(*) - returns 4000 >> nodetool status shows a single server at this point (using vnodes) >> >> In server2: >> made sure cassandra is not running >> sudo rm -rf /var/lib/cassandra/* >> Adjusted cassandra.yaml as above except that for seeds put both server >> addresses >> started cassandra >> waited a couple of min >> >> What we found: >> nodetool status on either server shows two servers, each with appox 50% >> (but not exactly) >> cqlsh>select count(*) - return 1870 (on either server) >> This process was repeated 3 times. each time the number was a bit >> different, but ~2000 >> >> Notes >> Replication factor is 1. >> No nodetool cleanup was run >> >> We have successfully added nodes in the past, but not since we moved to >> using vnodes >> THIS WAS A TEST. CLEAN MACHINES, SIMPLE DATA - What are we doing wrong? >> >> Thanks, >> >> Tamar Rosen >> Senior Data Architect >> Correlor.com >> >> >> >> >> >> >> > > --047d7b33d414b901e604ebb1c20c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
This worked, thanks.



On Thu, Nov 21, 2013 at= 5:20 PM, Julien Campan <julien.campan@gmail.com> wrot= e:

Hi,

You said : Adjusted cassandra.yaml as above except= that for seeds put both server addresses

= If you put the new node into the seeds list, y= our node will not bootstrap. This could explain why you see only the half of your data.

= Can you check in system.log ?


By the way, you can retry without adding the new server address in the seed= s list and normally=A0 should work :)

= You should add your new node into the seeds li= st only after the bootstrap operation.

= =A0

= Julien Campan

=A0


<= br>
2013/11/21 Tamar Rosen <= tamar@correlor.com<= /a>>
Hi,=

We are testing the process of adding a node to a cluster using a s= imple procedure, and seeing data loss.

System: Ubuntu 12.04 on AWS
Version: Cassandra + dsc 1.2.= 10

Here is what we did:
Created 2 new m1.large instances
=
Installed Java
Installed Cassandra 1.2.10 (the ve= rsion we are using in our production system)

In server1:<= br> Adjusted cassandra.yaml
=A0 comment out the initial_token
=A0 uncomment num_tokens: 256
=A0 changed "s= eeds" to the address of this server
=A0 changed listen_a= ddress to the address of this server
=A0 changed rpc_address to 0.0.0.0
=A0 changed pra= ctitioner to org.apache.cassandra.dht.RandomPartitioner
= made sure cassandra is not running
sudo rm -rf /var/lib/cassandra/= *
started cassandra
connected via cqlsh
Created a= new keyspace with replication factor 1
Created a new table
Populated the table with 4000 row of simple data using cql cop= y command
cqlsh> select count(*) - returns 4000
nodetool status show= s a single server at this point (using vnodes)

In server2:
made sure cassandra is not running
sudo rm -rf /var/lib/cassandra/*
Adjusted cassandra.yaml= as above except that for seeds put both server addresses
sta= rted cassandra
waited a couple of min

What we found:
nodetool status on either server shows two servers, each with appox 50% (bu= t not exactly)
cqlsh>select count(*) - return 1870 (on eit= her server)
This process was repeated 3 times. each time the = number was a bit different, but ~2000

Notes
Replication factor is 1.
No nodetool= cleanup was run

We have successfully added no= des in the past, but not since we moved to using vnodes
THIS WAS A TEST.= CLEAN MACHINES, SIMPLE DATA - What are we doing wrong?

Thanks,

Tamar Rosen
Senior = Data Architect
Correlor.com


=

=A0
=A0


--047d7b33d414b901e604ebb1c20c--