I'm setting up a new ring on 11 machines with cassandra 1.1.5. All seems to install fine and startup ok (from tarball) but I'm having an issue when updating the schema. There's one node that just doesn't want to receive the schema change. I've tried blowing away my /var/lib/cassandra/* directories and trying it a few times with the same results. I also tried the instructions here: http://wiki.apache.org/cassandra/FAQ#schema_disagreement
If I update the schema with cassandra-cli I get the following error:
Waiting for schema agreement...
The schema has not settled in 10 seconds; further migrations are ill-advised until it does.
Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199, 188.8.131.52, 184.108.40.206, 220.127.116.11, 172.19.102.135, 18.104.22.168, 22.214.171.124], b83fe28a-2851-34cb-bd8d-7d2621c1d872:[126.96.36.199]
then if I do a 'describe cluster' I get:
3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [188.8.131.52, 184.108.40.206, 220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199, 188.8.131.52, 184.108.40.206, 220.127.116.11, 18.104.22.168]
if I keep running describe cluster, both the schema UUID's(?) above keep changing every few seconds. Looking at the logs shows the nodes are continually doing a flush and compact.
if I start over by blowing away /var/lib/cassandra, and then try the schema update on the machine which is having trouble receiving it everything works ok and all nodes are in sync.
Anyone know what's going on here, or what I should look at to troubleshoot this?