Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.haddad@gmail.com
 designates 209.85.160.47 as permitted sender)
Sender: Jon Haddad <jonathan.haddad@gmail.com>
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: Upgrade from 1.0.9 to 1.2.8
From: Jon Haddad <jon@jonhaddad.com>
In-Reply-To: <5220C0D5.3030302@liquidweb.com>
Date: Fri, 30 Aug 2013 09:00:31 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <17715C2E-7368-4785-AF6D-B8857CF2FEAD@jonhaddad.com>
References: <5220C0D5.3030302@liquidweb.com>
To: user@cassandra.apache.org

Does your previous snapshot include the system keyspace?  I haven't =
tried upgrading from 1.0.x then rolling back, but it's possible there's =
some backwards incompatible changes.    Other than that, make sure you =
also rolled back your config files?=20

On Aug 30, 2013, at 8:57 AM, Mike Neir <mike@liquidweb.com> wrote:

> Greetings folks,
>=20
> I'm faced with the need to update a 36 node cluster with roughly 25T =
of data on disk to a version of cassandra in the 1.2.x series. While it =
seems that 1.2.8 will play nicely in the 1.0.9 cluster long enough to do =
a rolling upgrade, I'd still like to have a roll-back plan in case the =
rolling upgrade goes sideways.
>=20
> I've tried to upgrade a single node in my dev cluster, then roll back =
using a snapshot taken previously, but things don't appear to be going =
smoothly. The node will rejoin the ring eventually, but not after =
spending some time in the "Joining" state as shown by "nodetool ring", =
and spewing a ton of error messages similar to the following:
>=20
> ERROR [MutationStage:31] 2013-08-29 14:07:20,530 =
RowMutationVerbHandler.java (line 61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find =
cfId=3D1178
>=20
> My test procedure is as follows:
> 1)  nodetool -h localhost snapshot
> 2)  nodetool -h localhost drain
> 3)  service cassandra stop
> 4)  back up cassandra configs
> 5)  remove cassandra 1.0.9
> 6)  install cassandra 1.2.8
> 7)  restore cassandra configs, alter them to remove configuration =
entries no longer used
> 8)  start cassandra 1.2.8, let it run for a bit, then drain/stop it
> 9)  remove cassandra 1.2.8
> 10) reinstall cassandra 1.0.9
> 11) restore original cassandra configs
> 12) remove any commit logs present
> 13) remove folders for system_auth and system_traces Keyspaces (since =
they don't seem to be present in 1.0.9)
> 14) Move snapshots back to where they should be for 1.0.9 and remove =
cass 1.2.8 data
>  # cd /var/lib/cassandra/data/$KEYSPACE/
>  # mv */snapshots/$TIMESTAMP/* .
>  # find . -mindepth 1 -type d -exec rm -rf {} \;
>  # cd /var/lib/cassandra/data/system
>  # mv */snapshots/$TIMESTAMP/* .
>  # find . -mindepth 1 -type d -exec rm -rf {} \;
> 15) start cassandra 1.0.9
> 16) observe cassandra system.log
>=20
> Does anyone have any insight on things I may be doing wrong, or =
whether this is just an unavoidable pain point caused by rolling back? =
It seems that since there are no schema changes going on, the node =
should be able to just hop back into the cluster without error and =
without transitioning through the "Joining" state.
>=20
> --=20
>=20
>=20
>=20
> Mike Neir
> Liquid Web, Inc.
> Infrastructure Administrator
>=20