incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julien Campan <julien.cam...@gmail.com>
Subject Re: Simple test of adding a node causes data loss
Date Thu, 21 Nov 2013 15:20:37 GMT
Hi,

You said : Adjusted cassandra.yaml as above except that for seeds put both
server addresses

If you put the new node into the seeds list, your node will not bootstrap.
This could explain why you see only the half of your data.

Can you check in system.log ?


By the way, you can retry without adding the new server address in the
seeds list and normally  should work :)

You should add your new node into the seeds list only after the bootstrap
operation.



Julien Campan




2013/11/21 Tamar Rosen <tamar@correlor.com>

> Hi,
>
> We are testing the process of adding a node to a cluster using a simple
> procedure, and seeing data loss.
>
> System: Ubuntu 12.04 on AWS
> Version: Cassandra + dsc 1.2.10
>
> Here is what we did:
> Created 2 new m1.large instances
> Installed Java
> Installed Cassandra 1.2.10 (the version we are using in our production
> system)
>
> In server1:
> Adjusted cassandra.yaml
>   comment out the initial_token
>   uncomment num_tokens: 256
>   changed "seeds" to the address of this server
>   changed listen_address to the address of this server
>   changed rpc_address to 0.0.0.0
>   changed practitioner to org.apache.cassandra.dht.RandomPartitioner
> made sure cassandra is not running
> sudo rm -rf /var/lib/cassandra/*
> started cassandra
> connected via cqlsh
> Created a new keyspace with replication factor 1
> Created a new table
> Populated the table with 4000 row of simple data using cql copy command
> cqlsh> select count(*) - returns 4000
> nodetool status shows a single server at this point (using vnodes)
>
> In server2:
> made sure cassandra is not running
> sudo rm -rf /var/lib/cassandra/*
> Adjusted cassandra.yaml as above except that for seeds put both server
> addresses
> started cassandra
> waited a couple of min
>
> What we found:
> nodetool status on either server shows two servers, each with appox 50%
> (but not exactly)
> cqlsh>select count(*) - return 1870 (on either server)
> This process was repeated 3 times. each time the number was a bit
> different, but ~2000
>
> Notes
> Replication factor is 1.
> No nodetool cleanup was run
>
> We have successfully added nodes in the past, but not since we moved to
> using vnodes
> THIS WAS A TEST. CLEAN MACHINES, SIMPLE DATA - What are we doing wrong?
>
> Thanks,
>
> Tamar Rosen
> Senior Data Architect
> Correlor.com
>
>
>
>
>
>
>

Mime
View raw message