cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksey Yeschenko (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-11143) Schema changes don't propagate correctly if nodes are down
Date Thu, 11 Feb 2016 13:25:18 GMT


Aleksey Yeschenko commented on CASSANDRA-11143:

This is a direct consequence of CASSANDRA-5202; when you recreate that table, it gets a new
id (a timeuuid). When the downed node goes up and receives the delta, it's erroring out on
a mismatch.

This is a known issues and it will not go away any time soon - until we rewrite our schema
propagation code.

> Schema changes don't propagate correctly if nodes are down
> ----------------------------------------------------------
>                 Key: CASSANDRA-11143
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: PROD
>            Reporter: Anubhav Kale
> We saw a problem similar to what I describe below in our PROD environment a few times.
Below is a consistent repro. We can change the priority to Minor since there is a workaround,
> Using steps from,
setup a two node cluster locally. 
> . Bring up both nodes
> . Create a table, and ensure cqlsh is correctly showing it on both nodes.
> . Bring down one node
> . Drop and re-create the same table Or change some schema in the table.
> . Bring up the down node.
> You will notice the exceptions like below (because of schema mismatch), and the new schema
never propagates to this node that was down ((meaning  a select * via cqlsh will continue
to show old schema for the table). I let the cluster run for an hour to see if gossip will
somehow catch up. 
> However, the interesting part is if you restart this node that was down when schema changes
were made, the exception below goes away and it gets new schema correctly. 
> What is it caching that a second restart is necessary to make it behave correctly ?
> ERROR 00:23:33 Configuration exception merging remote schema
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found
7208d260-cf8c-11e5-a13b-fb6871b443fb; expected e2839010-cf7e-11e5-a13b-fb6871b443fb)
> 	at org.apache.cassandra.config.CFMetaData.validateCompatibility(
> 	at org.apache.cassandra.config.CFMetaData.apply( ~[main/:na]
> 	at org.apache.cassandra.config.Schema.updateTable( ~[main/:na]
> 	at org.apach

This message was sent by Atlassian JIRA

View raw message