Hey folks,

I have a 1.0.11 ring running in production with 6 nodes. Trying to bootstrap a new node in, and I'm getting the following consistently:

 INFO [main] 2012-09-05 04:24:13,317 StorageService.java (line 668) JOINING: waiting for schema information to complete


After waiting for over 30 minutes, I restarted the node to try again, and got the same thing. Tried wiping out the data dir on the new node, as well. Same result.

Turned on DEBUG, and got the following:

 INFO [main] 2012-09-05 03:58:55,205 StorageService.java (line 668) JOINING: waiting for schema information to complete
DEBUG [MigrationStage:1] 2012-09-05 03:59:11,440 DefinitionsUpdateVerbHandler.java (line 70) Applying UpdateColumnFamily from /10.140.128.218
DEBUG [MigrationStage:1] 2012-09-05 03:59:11,440 DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous version mismatch. cannot apply.
DEBUG [MigrationStage:1] 2012-09-05 03:59:11,631 DefinitionsUpdateVerbHandler.java (line 70) Applying UpdateColumnFamily from /10.140.128.218
DEBUG [MigrationStage:1] 2012-09-05 03:59:11,631 DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous version mismatch. cannot apply.


The logs continue with a bunch of failed migration errors from each node in the ring.

So, I'm guessing that there is a schema history problem on one of my nodes? Any clues on how I can fix this? I had considered wiping out the schema on one of my running nodes and starting it back up, but I'm worried it might not come back if it gets the same errors.


Also as a random question: is there any way to 'merge' historical schema changes together?


Thanks,
Jason