incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Langston, Jim" <>
Subject Re: alter column family ?
Date Thu, 11 Jul 2013 16:17:10 GMT
Hi Rob,

Are the schema's held somewhere else ? Going through the
process that you sent, when I restart the nodes, the original
schema's show up (btw, you were correct on your assessment,
even though the schema shows they are the same with the
gossipinfo command, they are not the same when looking
at them with cassandra-cli, not even close on 2 of the nodes).
So, I went through the process of clearing out the system CF's,
in steps 4 and 5, when the cassandra's restarted two of them (the
ones with the incorrect schema's), complained about the schema
and loaded what looks like a generic one. But, all of them have
schemas and 2 are correct and one is not.

This means I cannot execute step 7 , since the schema now exists
with the name on all the nodes. For example, the incorrect schema is
called MySchema, after the restart and the messages complaining
about CF's not existing, there is a schema called MySchema, on 2 nodes
they are correct, on 2 nodes they are not.

I have also tried to force the node with the incorrect schema to come
up on its own by shutting down the cluster except for a node with a
correct schema. I went through the same steps and brought that
node down and back up, same results.

Thoughts ? ideas ?


From: Robert Coli <<>>
Reply-To: <<>>
Date: Tue, 9 Jul 2013 17:10:53 -0700
To: "<>" <<>>
Subject: Re: alter column family ?

On Tue, Jul 9, 2013 at 11:52 AM, Langston, Jim <<>>
> On the command (4 node cluster):
> nodetool gossipinfo -h localhost |grep SCHEMA |sort | uniq -c | sort -n
>       4   SCHEMA:60edeaa8-70a4-3825-90a5-d7746ffa8e4d

If your schemas actually agree (and given that you're in 1.1.2) you probably are encountering

Which is one of the 1.1.2 era "schema stuck" issues I was referring to earlier.

> On the second part, I have the same Cassandra version in staging and
> production, with staging being a smaller cluster. Not sure what you mean
> by nuking schema's (ie. delete directories ?)

I like when googling things returns related threads in which I have previously advised people
to do a detailed list of things, heh :

Here's a slightly clarified version of these steps...

0. Dump your existing schema to schema_definition_file
1. Take all nodes out of service;
2. Run nodetool drain on each and verify that they have drained (grep -i DRAINED system.log)
3. Stop cassandra on each node;
4. Move /var/lib/cassandra/data/system out of the way
5. Move /var/lib/cassandra/saved_caches/system-* out of the way
6. Start all nodes;
7. cassandra-cli < schema_definition_file on one node only. (includes
create keyspace and create column familiy entries)

Note: you should not literally do this, you should break your schema_definition_file into
individual statements and wait until schema agreement between each DDL statement.

8. Put the nodes back in service.
9. Done.


View raw message