cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabien Rousseau <fab...@yakaz.com>
Subject Re: Problems after trying a migration
Date Wed, 18 Mar 2015 16:31:43 GMT
Hi David,

There is an excellent article which describes exactly what you want to do
(ie migrate from one DC to another DC) :
http://planetcassandra.org/blog/cassandra-migration-to-ec2/

2015-03-18 17:05 GMT+01:00 David CHARBONNIER <David.CHARBONNIER@rgsystem.com
>:

>  Hi,
>
>
>
> We’re using Cassandra through the Datastax Enterprise package in version
> 4.5.1 (Cassandra version 2.0.8.39) with 7 nodes in a single datacenter.
>
>
>
> We need to move our Cassandra cluster from France to another country. To
> do this, we want to add a second 7-nodes datacenter to our cluster and
> stream all data between the two countries before dropping the first
> datacenter.
>
>
>
> On January 31st, we tried doing so but we had some problems:
>
> -          New nodes in the other country have been installed like French
> nodes except for Datastax Enterprise version (4.5.1 in France and 4.6.1 in
> the other country which means Cassandra version 2.0.8.39 in France and
> 2.0.12.200 in the other country)
>
> -          The following procedure has been followed:
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
> but an error occurred during step 3. New nodes have been started before
> the *cassandra-topology.properties* file has been updated on the original
> datacenter. New nodes appeared in the original datacenter instead of the
> new one.
>
> -          To recover our original cluster, we decommissionned every node
> of the new datacenter with the *nodetool decommission* command.
>
>
>
> On February 9th, nodes in the second datacenter have been restarted and
> joined the cluster. We had to decommission them just like before.
>
>
>
> On February 11th, we added disk space on our 7 running French nodes. To
> achieve this, we restarted the cluster but the nodes updated their perring
> informations and nodes from Luxembourg (decommissionned on February 9th)
> were present. This behaviour is described here:
> https://issues.apache.org/jira/browse/CASSANDRA-7825. So we cleaned
> *system.peers* table content.
>
>
>
> On March 11th, we needed to add an 8th node to our existing French
> cluster. We installed the same Datastax Enterprise version (4.5.1 with
> Cassandra 2.0.8.39) and tried to add this node to the cluster with this
> procedure:
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html.
> In OPSCenter, the node was joining the cluster and data streaming got stuck
> at 100%. After several hours, *nodetool status* showed us that the node
> was still joining but nothing in the logs let us know there was a problem. We
> restarted the node but it has no effect. Then we cleaned data and commitlog
> contents and try to add the node to the cluster again but without result.
>
> Last try was to add the node with *auto_bootstrap : false* in order to
> add the node to the cluster manually but it messed up with the data. So we
> shut down the node and decommissioned it (with *nodetool removenode*).
> The whole cluster has been repaired and we stopped doing anything.
>
>
>
> Now, our cluster has only 7 French nodes in which we can’t add any node. The
> OPSCenter data has disapeared and we work without any information about how
> our cluster is running.
>
>
>
> You’ll find attached to this email our current configuration and a
> screenshot of our OPSCenter metric page.
>
>
>
> Do you have some idea on how to clean up the mess and get our cluster
> running cleanly before we start our migration (France to another country
> like described in the beginning of this email)?
>
>
>
> Thank you.
>
>
>
> Best regards,
>
>
>
>     *David CHARBONNIER*
>
> Sysadmin
>
> T : +33 411 934 200
>
> david.charbonnier@rgsystem.com
>
> ZAC Aéroport
>
> 125 Impasse Adam Smith
>
> 34470 Pérols - France
>
> *www.rgsystem.com* <http://www.rgsystem.com/>
>
>
>
>
>
>
>



-- 
Fabien Rousseau


 <aurore@yakaz.com>www.yakaz.com

Mime
View raw message