incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Rosen <ta...@correlor.com>
Subject Simple test of adding a node causes data loss
Date Thu, 21 Nov 2013 14:51:49 GMT
Hi,

We are testing the process of adding a node to a cluster using a simple
procedure, and seeing data loss.

System: Ubuntu 12.04 on AWS
Version: Cassandra + dsc 1.2.10

Here is what we did:
Created 2 new m1.large instances
Installed Java
Installed Cassandra 1.2.10 (the version we are using in our production
system)

In server1:
Adjusted cassandra.yaml
  comment out the initial_token
  uncomment num_tokens: 256
  changed "seeds" to the address of this server
  changed listen_address to the address of this server
  changed rpc_address to 0.0.0.0
  changed practitioner to org.apache.cassandra.dht.RandomPartitioner
made sure cassandra is not running
sudo rm -rf /var/lib/cassandra/*
started cassandra
connected via cqlsh
Created a new keyspace with replication factor 1
Created a new table
Populated the table with 4000 row of simple data using cql copy command
cqlsh> select count(*) - returns 4000
nodetool status shows a single server at this point (using vnodes)

In server2:
made sure cassandra is not running
sudo rm -rf /var/lib/cassandra/*
Adjusted cassandra.yaml as above except that for seeds put both server
addresses
started cassandra
waited a couple of min

What we found:
nodetool status on either server shows two servers, each with appox 50%
(but not exactly)
cqlsh>select count(*) - return 1870 (on either server)
This process was repeated 3 times. each time the number was a bit
different, but ~2000

Notes
Replication factor is 1.
No nodetool cleanup was run

We have successfully added nodes in the past, but not since we moved to
using vnodes
THIS WAS A TEST. CLEAN MACHINES, SIMPLE DATA - What are we doing wrong?

Thanks,

Tamar Rosen
Senior Data Architect
Correlor.com

Mime
View raw message