cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Harvey <alie...@gmail.com>
Subject Re: Cannot bootstrap new nodes in 1.0.11 ring - schema issue
Date Fri, 07 Sep 2012 11:52:06 GMT
After a chat with driftx today, I tried wiping out my MigrationInfo on the 
ring and rolling a restart. I then made a single change to the schema so at 
least 1 migration would exist. Unfortunately the same error persists: 
"Previous version mismatch". Also occasionally the node is bootstrapping 
without applying any schema on startup. The behaviour is inconsistent, 
despite wiping the entire data directory and commitlogs on the 
bootstrapping node.

I added some debug statements to Migration.java to find exactly what the 
mismatch was. Here is what I have now:

DEBUG [MigrationStage:1] 2012-09-07 04:44:53,669 Migration.java (line 98) 
lastversion: ee323110-eedf-11e1-0000-5027269873df  getVersion: 
00000000-0000-1000-0000-000000000000
DEBUG [MigrationStage:1] 2012-09-07 04:44:53,669 
DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous 
version mismatch. cannot apply.
DEBUG [MigrationStage:1] 2012-09-07 04:44:53,670 
DefinitionsUpdateVerbHandler.java (line 70) Applying UpdateColumnFamily 
from /10.140.129.18
DEBUG [MigrationStage:1] 2012-09-07 04:44:53,670 Migration.java (line 98) 
lastversion: ee323110-eedf-11e1-0000-5027269873df  getVersion: 
00000000-0000-1000-0000-000000000000
DEBUG [MigrationStage:1] 2012-09-07 04:44:53,670 
DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous 
version mismatch. cannot apply.

The "Previous version mismatch" event happens when lastVersion != 
getVersion. Obviously that is the case here, as getVersion is blank. Don't 
all nodes bootstrap with a blank schema version? Why would the Migration 
logic expect the lastVersion to match the bootstrapping nodes getVersion?



On Wednesday, September 5, 2012 4:29:34 AM UTC-7, Jason Harvey wrote:
>
> Hey folks,
>
> I have a 1.0.11 ring running in production with 6 nodes. Trying to 
> bootstrap a new node in, and I'm getting the following consistently:
>
>  INFO [main] 2012-09-05 04:24:13,317 StorageService.java (line 668) 
> JOINING: waiting for schema information to complete
>
>
> After waiting for over 30 minutes, I restarted the node to try again, and 
> got the same thing. Tried wiping out the data dir on the new node, as well. 
> Same result.
>
> Turned on DEBUG, and got the following:
>
>  INFO [main] 2012-09-05 03:58:55,205 StorageService.java (line 668) 
> JOINING: waiting for schema information to complete
> DEBUG [MigrationStage:1] 2012-09-05 03:59:11,440 
> DefinitionsUpdateVerbHandler.java (line 70) Applying UpdateColumnFamily 
> from /10.140.128.218
> DEBUG [MigrationStage:1] 2012-09-05 03:59:11,440 
> DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous 
> version mismatch. cannot apply.
> DEBUG [MigrationStage:1] 2012-09-05 03:59:11,631 
> DefinitionsUpdateVerbHandler.java (line 70) Applying UpdateColumnFamily 
> from /10.140.128.218
> DEBUG [MigrationStage:1] 2012-09-05 03:59:11,631 
> DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous 
> version mismatch. cannot apply.
>
>
> The logs continue with a bunch of failed migration errors from each node 
> in the ring.
>
> So, I'm guessing that there is a schema history problem on one of my 
> nodes? Any clues on how I can fix this? I had considered wiping out the 
> schema on one of my running nodes and starting it back up, but I'm worried 
> it might not come back if it gets the same errors.
>
>
> Also as a random question: is there any way to 'merge' historical schema 
> changes together?
>
>
> Thanks,
> Jason
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message