incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Haddad <...@jonhaddad.com>
Subject Re: Upgrade from 1.0.9 to 1.2.8
Date Fri, 30 Aug 2013 16:01:13 GMT
Sorry, I didn't see the test procedure, it's still early.

On Aug 30, 2013, at 8:57 AM, Mike Neir <mike@liquidweb.com> wrote:

> Greetings folks,
> 
> I'm faced with the need to update a 36 node cluster with roughly 25T of data on disk
to a version of cassandra in the 1.2.x series. While it seems that 1.2.8 will play nicely
in the 1.0.9 cluster long enough to do a rolling upgrade, I'd still like to have a roll-back
plan in case the rolling upgrade goes sideways.
> 
> I've tried to upgrade a single node in my dev cluster, then roll back using a snapshot
taken previously, but things don't appear to be going smoothly. The node will rejoin the ring
eventually, but not after spending some time in the "Joining" state as shown by "nodetool
ring", and spewing a ton of error messages similar to the following:
> 
> ERROR [MutationStage:31] 2013-08-29 14:07:20,530 RowMutationVerbHandler.java (line 61)
Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1178
> 
> My test procedure is as follows:
> 1)  nodetool -h localhost snapshot
> 2)  nodetool -h localhost drain
> 3)  service cassandra stop
> 4)  back up cassandra configs
> 5)  remove cassandra 1.0.9
> 6)  install cassandra 1.2.8
> 7)  restore cassandra configs, alter them to remove configuration entries no longer used
> 8)  start cassandra 1.2.8, let it run for a bit, then drain/stop it
> 9)  remove cassandra 1.2.8
> 10) reinstall cassandra 1.0.9
> 11) restore original cassandra configs
> 12) remove any commit logs present
> 13) remove folders for system_auth and system_traces Keyspaces (since they don't seem
to be present in 1.0.9)
> 14) Move snapshots back to where they should be for 1.0.9 and remove cass 1.2.8 data
>  # cd /var/lib/cassandra/data/$KEYSPACE/
>  # mv */snapshots/$TIMESTAMP/* .
>  # find . -mindepth 1 -type d -exec rm -rf {} \;
>  # cd /var/lib/cassandra/data/system
>  # mv */snapshots/$TIMESTAMP/* .
>  # find . -mindepth 1 -type d -exec rm -rf {} \;
> 15) start cassandra 1.0.9
> 16) observe cassandra system.log
> 
> Does anyone have any insight on things I may be doing wrong, or whether this is just
an unavoidable pain point caused by rolling back? It seems that since there are no schema
changes going on, the node should be able to just hop back into the cluster without error
and without transitioning through the "Joining" state.
> 
> -- 
> 
> 
> 
> Mike Neir
> Liquid Web, Inc.
> Infrastructure Administrator
> 


Mime
View raw message