incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Rosen <ta...@correlor.com>
Subject Re: Simple test of adding a node causes data loss
Date Thu, 21 Nov 2013 15:43:35 GMT
This worked, thanks.



On Thu, Nov 21, 2013 at 5:20 PM, Julien Campan <julien.campan@gmail.com>wrote:

> Hi,
>
> You said : Adjusted cassandra.yaml as above except that for seeds put both
> server addresses
>
> If you put the new node into the seeds list, your node will not bootstrap.
> This could explain why you see only the half of your data.
>
> Can you check in system.log ?
>
>
> By the way, you can retry without adding the new server address in the
> seeds list and normally  should work :)
>
> You should add your new node into the seeds list only after the bootstrap
> operation.
>
>
>
> Julien Campan
>
>
>
>
> 2013/11/21 Tamar Rosen <tamar@correlor.com>
>
>> Hi,
>>
>> We are testing the process of adding a node to a cluster using a simple
>> procedure, and seeing data loss.
>>
>> System: Ubuntu 12.04 on AWS
>> Version: Cassandra + dsc 1.2.10
>>
>> Here is what we did:
>> Created 2 new m1.large instances
>> Installed Java
>> Installed Cassandra 1.2.10 (the version we are using in our production
>> system)
>>
>> In server1:
>> Adjusted cassandra.yaml
>>   comment out the initial_token
>>   uncomment num_tokens: 256
>>   changed "seeds" to the address of this server
>>   changed listen_address to the address of this server
>>   changed rpc_address to 0.0.0.0
>>   changed practitioner to org.apache.cassandra.dht.RandomPartitioner
>> made sure cassandra is not running
>> sudo rm -rf /var/lib/cassandra/*
>> started cassandra
>> connected via cqlsh
>> Created a new keyspace with replication factor 1
>> Created a new table
>> Populated the table with 4000 row of simple data using cql copy command
>> cqlsh> select count(*) - returns 4000
>> nodetool status shows a single server at this point (using vnodes)
>>
>> In server2:
>> made sure cassandra is not running
>> sudo rm -rf /var/lib/cassandra/*
>> Adjusted cassandra.yaml as above except that for seeds put both server
>> addresses
>> started cassandra
>> waited a couple of min
>>
>> What we found:
>> nodetool status on either server shows two servers, each with appox 50%
>> (but not exactly)
>> cqlsh>select count(*) - return 1870 (on either server)
>> This process was repeated 3 times. each time the number was a bit
>> different, but ~2000
>>
>> Notes
>> Replication factor is 1.
>> No nodetool cleanup was run
>>
>> We have successfully added nodes in the past, but not since we moved to
>> using vnodes
>> THIS WAS A TEST. CLEAN MACHINES, SIMPLE DATA - What are we doing wrong?
>>
>> Thanks,
>>
>> Tamar Rosen
>> Senior Data Architect
>> Correlor.com
>>
>>
>>
>>
>>
>>
>>
>
>

Mime
View raw message