incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Neir <>
Subject Upgrade from 1.0.9 to 1.2.8
Date Fri, 30 Aug 2013 15:57:09 GMT
Greetings folks,

I'm faced with the need to update a 36 node cluster with roughly 25T of data on 
disk to a version of cassandra in the 1.2.x series. While it seems that 1.2.8 
will play nicely in the 1.0.9 cluster long enough to do a rolling upgrade, I'd 
still like to have a roll-back plan in case the rolling upgrade goes sideways.

I've tried to upgrade a single node in my dev cluster, then roll back using a 
snapshot taken previously, but things don't appear to be going smoothly. The 
node will rejoin the ring eventually, but not after spending some time in the 
"Joining" state as shown by "nodetool ring", and spewing a ton of error messages 
similar to the following:

ERROR [MutationStage:31] 2013-08-29 14:07:20,530 
(line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1178

My test procedure is as follows:
1)  nodetool -h localhost snapshot
2)  nodetool -h localhost drain
3)  service cassandra stop
4)  back up cassandra configs
5)  remove cassandra 1.0.9
6)  install cassandra 1.2.8
7)  restore cassandra configs, alter them to remove configuration entries no 
longer used
8)  start cassandra 1.2.8, let it run for a bit, then drain/stop it
9)  remove cassandra 1.2.8
10) reinstall cassandra 1.0.9
11) restore original cassandra configs
12) remove any commit logs present
13) remove folders for system_auth and system_traces Keyspaces (since they don't 
seem to be present in 1.0.9)
14) Move snapshots back to where they should be for 1.0.9 and remove cass 1.2.8 data
   # cd /var/lib/cassandra/data/$KEYSPACE/
   # mv */snapshots/$TIMESTAMP/* .
   # find . -mindepth 1 -type d -exec rm -rf {} \;
   # cd /var/lib/cassandra/data/system
   # mv */snapshots/$TIMESTAMP/* .
   # find . -mindepth 1 -type d -exec rm -rf {} \;
15) start cassandra 1.0.9
16) observe cassandra system.log

Does anyone have any insight on things I may be doing wrong, or whether this is 
just an unavoidable pain point caused by rolling back? It seems that since there 
are no schema changes going on, the node should be able to just hop back into 
the cluster without error and without transitioning through the "Joining" state.


Mike Neir
Liquid Web, Inc.
Infrastructure Administrator

View raw message