incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: alter column family ?
Date Tue, 09 Jul 2013 18:35:35 GMT
On Tue, Jul 9, 2013 at 10:26 AM, Robert Coli <rcoli@eventbrite.com> wrote:

> nodetool -h localhost netstats |grep SCHEMA |sort | uniq -c | sort -n
>

Sorry, I meant "gossipinfo" and not "netstats".

With the right command, do you see that all nodes in the cluster have the
same schema version?


> I'm on version 1.1.2
>

1) Hinted Handoff is broken in 1.1.2, upgrade ASAP.
2) I believe the particular case you are encountering may be a more
specific bug from the 1.1.2 timeframe.
3) Desynched schema like you seem to be encountering is very common in
1.1.2 timeframe. In most cases the best/only solution is :
   a) drain all nodes, and stop them
   b) nuke schema on all nodes (optionally nuke/move aside entire system
keyspace)
   c) start nodes, waiting for cluster to completely coalesce
   d) re-load schema one statement at a time, BEING SURE TO WAIT FOR SCHEMA
AGREEMENT ON ***ALL NODES*** before running the next schema altering
statement

If you are unable to take an outage on this cluster, there are other ways
to resolve issues like this but generally they will both be complex and
error prone and will take much more time and effort than doing the above.

=Rob

Mime
View raw message