Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9F690200D23 for ; Thu, 5 Oct 2017 02:00:33 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9D774160BD7; Thu, 5 Oct 2017 00:00:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9B3F01609DD for ; Thu, 5 Oct 2017 02:00:31 +0200 (CEST) Received: (qmail 93922 invoked by uid 500); 5 Oct 2017 00:00:30 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 93911 invoked by uid 99); 5 Oct 2017 00:00:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Oct 2017 00:00:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BBE6FC82A6 for ; Thu, 5 Oct 2017 00:00:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 89Gd-JqNzR6h for ; Thu, 5 Oct 2017 00:00:25 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 7DF9F5FAEA for ; Thu, 5 Oct 2017 00:00:24 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8A619E06BA for ; Thu, 5 Oct 2017 00:00:23 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 3B08C24301 for ; Thu, 5 Oct 2017 00:00:23 +0000 (UTC) Date: Thu, 5 Oct 2017 00:00:23 +0000 (UTC) From: "Jay Zhuang (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (CASSANDRA-8387) Schema inconsistency (cached vs schema_columnfamilies) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 05 Oct 2017 00:00:33 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-8387?page=3Dcom.atlas= sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D= 16192225#comment-16192225 ]=20 Jay Zhuang edited comment on CASSANDRA-8387 at 10/4/17 11:59 PM: ----------------------------------------------------------------- Is there any mitigation other than rolling restart the cluster? We saw the = problem in multiple clusters (version 3.0.14), it's causing other table cre= ation returns failure, for example, when create table {{tt81}}, it returns = {{tt4}} has problem: {noformat} ./bin/cqlsh 127.0.0.1 -e "create table zjay_test.tt81(tt int PRIMARY KEY, n= ame text, id text)" :1:ServerError: java.lang.RuntimeException: java.util.concurrent.Exe= cutionException: org.apache.cassandra.exceptions.ConfigurationException: Co= lumn family ID mismatch for keyspace zjay_test table tt4 (found 8f64bab0-a8= 9b-11e7-bca7-8d4000ea9db6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) {noformat} The table {{tt81}} is successfully created but the schema id is never chang= ed, I guess it's because the migration task failed: {noformat} DEBUG [MigrationStage:1] 2017-10-04 11:44:43,185 Schema.java:465 - Adding o= rg.apache.cassandra.config.CFMetaData@540ff3c8[cfId=3D1605ec90-a934-11e7-b0= 3f-237cabe3739e,ksName=3Dzjay_test,cfName=3Dtt81,flags=3D[COMPOUND],params= =3DTableParams{comment=3D, read_repair_chance=3D0.0, dclocal_read_repair_ chance=3D0.1, bloom_filter_fp_chance=3D0.01, crc_check_chance=3D1.0, gc_gra= ce_seconds=3D864000, default_time_to_live=3D0, memtable_flush_period_in_ms= =3D0, min_index_interval=3D128, max_index_interval=3D2048, speculative_retr= y=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_per_partition' : 'NONE'}= , co mpaction=3DCompactionParams{class=3Dorg.apache.cassandra.db.compaction.Size= TieredCompactionStrategy, options=3D{max_threshold=3D32, min_threshold=3D4}= }, compression=3Dorg.apache.cassandra.schema.CompressionParams@78dfc45f, ex= tensions=3D{}, cdc=3Dfalse},comparator=3Dcomparator(),partitionColumns=3D[[= ] | [id name]],partitionKeyColumns=3D[ColumnDefinition{name=3Dtt, type=3Dorg.a= pache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}],= clusteringColumns=3D[],keyValidator=3Dorg.apache.cassandra.db.marshal.Int32= Type,columnMetadata=3D[ColumnDefinition{name=3Dtt, type=3Dorg.apache.cassan= dra .db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}, ColumnDefinitio= n{name=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DREGULA= R, position=3D-1}, ColumnDefinition{name=3Dname, type=3Dorg.apache.cassandr= a.db.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}],droppedColumns=3D{},= trigg ers=3D[],indexes=3D[]] to cfIdMap DEBUG [MigrationStage:1] 2017-10-04 11:44:43,186 CFMetaData.java:745 - appl= ying org.apache.cassandra.config.CFMetaData@6e7e6efb[cfId=3D8f64bab0-a89b-1= 1e7-bca7-8d4000ea9db6,ksName=3Dzjay_test,cfName=3Dtt4,flags=3D[COMPOUND],pa= rams=3DTableParams{comment=3D, read_repair_chance=3D0.0, dclocal_read_re pair_chance=3D0.1, bloom_filter_fp_chance=3D0.01, crc_check_chance=3D1.0, g= c_grace_seconds=3D864000, default_time_to_live=3D0, memtable_flush_period_i= n_ms=3D0, min_index_interval=3D128, max_index_interval=3D2048, speculative_= retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_per_partition' : 'NO= NE' }, compaction=3DCompactionParams{class=3Dorg.apache.cassandra.db.compaction= .SizeTieredCompactionStrategy, options=3D{max_threshold=3D32, min_threshold= =3D4}}, compression=3Dorg.apache.cassandra.schema.CompressionParams@78dfc45= f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcomparator(),partitionColumns= =3D [[] | [id name]],partitionKeyColumns=3D[ColumnDefinition{name=3Dtt, type=3D= org.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position= =3D0}],clusteringColumns=3D[],keyValidator=3Dorg.apache.cassandra.db.marsha= l.Int32Type,columnMetadata=3D[ColumnDefinition{name=3Dtt, type=3Dorg.apache= .cass andra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}, ColumnDefi= nition{name=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DR= EGULAR, position=3D-1}, ColumnDefinition{name=3Dname, type=3Dorg.apache.cas= sandra.db.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}],droppedColumns= =3D{}, triggers=3D[],indexes=3D[]] to org.apache.cassandra.config.CFMetaData@69e66= 630[cfId=3D8f5ccb70-a89b-11e7-b03f-237cabe3739e,ksName=3Dzjay_test,cfName= =3Dtt4,flags=3D[COMPOUND],params=3DTableParams{comment=3D, read_repair_chan= ce=3D0.0, dclocal_read_repair_chance=3D0.1, bloom_filter_fp_chance=3D0.01, = crc_che ck_chance=3D1.0, gc_grace_seconds=3D864000, default_time_to_live=3D0, memta= ble_flush_period_in_ms=3D0, min_index_interval=3D128, max_index_interval=3D= 2048, speculative_retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_pe= r_partition' : 'NONE'}, compaction=3DCompactionParams{class=3Dorg.apache.ca= ssa ndra.db.compaction.SizeTieredCompactionStrategy, options=3D{max_threshold= =3D32, min_threshold=3D4}}, compression=3Dorg.apache.cassandra.schema.Compr= essionParams@78dfc45f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcomparato= r(),partitionColumns=3D[[] | [id name]],partitionKeyColumns=3D[ColumnDefini= tio n{name=3Dtt, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTI= TION_KEY, position=3D0}],clusteringColumns=3D[],keyValidator=3Dorg.apache.c= assandra.db.marshal.Int32Type,columnMetadata=3D[ColumnDefinition{name=3Dtt,= type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, po= sit ion=3D0}, ColumnDefinition{name=3Did, type=3Dorg.apache.cassandra.db.marsha= l.UTF8Type, kind=3DREGULAR, position=3D-1}, ColumnDefinition{name=3Dname, t= ype=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DREGULAR, position=3D= -1}],droppedColumns=3D{},triggers=3D[],indexes=3D[]] ERROR [MigrationStage:1] 2017-10-04 11:44:43,187 CassandraDaemon.java:207 -= Exception in thread Thread[MigrationStage:1,5,main] org.apache.cassandra.exceptions.ConfigurationException: Column family ID mi= smatch for keyspace zjay_test table tt4 (found 8f64bab0-a89b-11e7-bca7-8d40= 00ea9db6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaD= ata.java:785) ~[main/:na] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[= main/:na] at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[ma= in/:na] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeys= pace.java:1395) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspac= e.java:1351) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVer= sion(SchemaKeyspace.java:1301) ~[main/:na] at org.apache.cassandra.service.MigrationManager$1.runMayThrow(Migratio= nManager.java:543) ~[main/:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:= 28) ~[main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) ~[na:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_= 141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) ~[na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) [na:1.8.0_141] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca= lDeallocator$0(NamedThreadFactory.java:79) [main/:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141] ERROR [SharedPool-Worker-7] 2017-10-04 11:44:43,188 QueryMessage.java:128 -= Unexpected error during query java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.ap= ache.cassandra.exceptions.ConfigurationException: Column family ID mismatch= for keyspace zjay_test table tt4 (found 8f64bab0-a89b-11e7-bca7-8d4000ea9d= b6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java= :402) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announce(MigrationMana= ger.java:525) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamil= y(MigrationManager.java:355) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamil= y(MigrationManager.java:340) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamil= y(MigrationManager.java:320) ~[main/:na] at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMi= gration(CreateTableStatement.java:87) ~[main/:na] at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute= (SchemaAlteringStatement.java:93) ~[main/:na] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProce= ssor.java:206) ~[main/:na] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java= :237) ~[main/:na] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java= :222) ~[main/:na] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMe= ssage.java:115) ~[main/:na] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Messa= ge.java:513) [main/:na] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Messa= ge.java:407) [main/:na] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChann= elInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abs= tractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Fin= al] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractCh= annelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannel= HandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) [na:1.8.0_141] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$Fu= tureTask.run(AbstractLocalAwareExecutorService.java:164) [main/:na] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [m= ain/:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141] {noformat} Schema push also failed on other nodes: {noformat} DEBUG [InternalResponseStage:19] 2017-10-04 14:52:20,324 CFMetaData.java:74= 5 - applying org.apache.cassandra.config.CFMetaData@6c71d136[cfId=3D8f64bab= 0-a89b-11e7-bca7-8d4000ea9db6,ksName=3Dzjay_test,cfName=3Dtt4,flags=3D[COMP= OUND],params=3DTableParams{comment=3D, read_repair_chance=3D0.0, dclocal _read_repair_chance=3D0.1, bloom_filter_fp_chance=3D0.01, crc_check_chance= =3D1.0, gc_grace_seconds=3D864000, default_time_to_live=3D0, memtable_flush= _period_in_ms=3D0, min_index_interval=3D128, max_index_interval=3D2048, spe= culative_retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_per_partiti= on' : 'NONE'}, compaction=3DCompactionParams{class=3Dorg.apache.cassandra.db.co= mpaction.SizeTieredCompactionStrategy, options=3D{max_threshold=3D32, min_t= hreshold=3D4}}, compression=3Dorg.apache.cassandra.schema.CompressionParams= @78dfc45f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcomparator(),partitio= n Columns=3D[[] | [id name]],partitionKeyColumns=3D[ColumnDefinition{name=3Dt= t, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, = position=3D0}],clusteringColumns=3D[],keyValidator=3Dorg.apache.cassandra.d= b.marshal.Int32Type,columnMetadata=3D[ColumnDefinition{name=3Dtt, type=3Dor= g.apa che.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}, Co= lumnDefinition{name=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, = kind=3DREGULAR, position=3D-1}, ColumnDefinition{name=3Dname, type=3Dorg.ap= ache.cassandra.db.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}],dropped= Col umns=3D{},triggers=3D[],indexes=3D[]] to org.apache.cassandra.config.CFMeta= Data@24fcccbc[cfId=3D8f5ccb70-a89b-11e7-b03f-237cabe3739e,ksName=3Dzjay_tes= t,cfName=3Dtt4,flags=3D[COMPOUND],params=3DTableParams{comment=3D, read_rep= air_chance=3D0.0, dclocal_read_repair_chance=3D0.1, bloom_filter_fp_chance= =3D0.01, crc_check_chance=3D1.0, gc_grace_seconds=3D864000, default_time_to_live=3D= 0, memtable_flush_period_in_ms=3D0, min_index_interval=3D128, max_index_int= erval=3D2048, speculative_retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', = 'rows_per_partition' : 'NONE'}, compaction=3DCompactionParams{class=3Dorg.a= pac he.cassandra.db.compaction.SizeTieredCompactionStrategy, options=3D{max_thr= eshold=3D32, min_threshold=3D4}}, compression=3Dorg.apache.cassandra.schema= .CompressionParams@78dfc45f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcom= parator(),partitionColumns=3D[[] | [id name]],partitionKeyColumns=3D[Column= D efinition{name=3Dtt, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind= =3DPARTITION_KEY, position=3D0}],clusteringColumns=3D[],keyValidator=3Dorg.= apache.cassandra.db.marshal.Int32Type,columnMetadata=3D[ColumnDefinition{na= me=3Dtt, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION= _KE Y, position=3D0}, ColumnDefinition{name=3Did, type=3Dorg.apache.cassandra.d= b.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}, ColumnDefinition{name= =3Dname, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DREGULAR, p= osition=3D-1}],droppedColumns=3D{},triggers=3D[],indexes=3D[]] ERROR [InternalResponseStage:19] 2017-10-04 14:52:20,325 MigrationTask.java= :96 - Configuration exception merging remote schema org.apache.cassandra.exceptions.ConfigurationException: Column family ID mi= smatch for keyspace zjay_test table tt4 (found 8f64bab0-a89b-11e7-bca7-8d40= 00ea9db6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaD= ata.java:785) ~[main/:na] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[= main/:na] at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[ma= in/:na] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeys= pace.java:1395) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspac= e.java:1351) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVer= sion(SchemaKeyspace.java:1301) ~[main/:na] at org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.= java:92) ~[main/:na] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHand= ler.java:53) [main/:na] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask= .java:67) [main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) [na:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_1= 41] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) [na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) [na:1.8.0_141] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca= lDeallocator$0(NamedThreadFactory.java:79) [main/:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141] {noformat} I tried `nodetool resetlocalschema`, which causes following failure: {noformat} INFO [MigrationStage:1] 2017-10-04 14:50:51,677 ColumnFamilyStore.java:389= - Initializing zjay_test.tt4 ERROR [MigrationStage:1] 2017-10-04 14:50:51,679 CassandraDaemon.java:207 -= Exception in thread Thread[MigrationStage:1,5,main] java.lang.RuntimeException: javax.management.InstanceAlreadyExistsException= : org.apache.cassandra.db:type=3DTables,keyspace=3Dzjay_test,table=3Dtt4 at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.j= ava:439) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.j= ava:361) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(Co= lumnFamilyStore.java:577) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(Co= lumnFamilyStore.java:554) ~[main/:na] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:368) ~[main/:n= a] at org.apache.cassandra.config.Schema.addTable(Schema.java:649) ~[main/= :na] at java.util.HashMap$Values.forEach(HashMap.java:980) ~[na:1.8.0_141] at java.util.Collections$UnmodifiableCollection.forEach(Collections.jav= a:1080) ~[na:1.8.0_141] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeys= pace.java:1386) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspac= e.java:1351) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVer= sion(SchemaKeyspace.java:1301) ~[main/:na] at org.apache.cassandra.service.MigrationManager$1.runMayThrow(Migratio= nManager.java:543) ~[main/:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:= 28) ~[main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) ~[na:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_= 141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) ~[na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) [na:1.8.0_141] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca= lDeallocator$0(NamedThreadFactory.java:79) [main/:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141] Caused by: javax.management.InstanceAlreadyExistsException: org.apache.cass= andra.db:type=3DTables,keyspace=3Dzjay_test,table=3Dtt4 at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) ~[n= a:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRe= pository(DefaultMBeanServerInterceptor.java:1898) ~[na:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynami= cMBean(DefaultMBeanServerInterceptor.java:966) ~[na:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject= (DefaultMBeanServerInterceptor.java:900) ~[na:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(= DefaultMBeanServerInterceptor.java:324) ~[na:1.8.0_141] at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.= java:522) ~[na:1.8.0_141] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.j= ava:434) ~[main/:na] ... 18 common frames omitted {noformat} and inconsistent schema ID: {noformat} Cluster Information: Name: tt2 Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: 9a47fb8d-1712-362e-b422-6331fe448d10: [127.0.0.3] 776aa76b-1c7a-3bf8-9891-4564b72981eb: [127.0.0.1] 59adb24e-f3cd-3e02-97f0-5b395827453f: [127.0.0.2] {noformat} Here is the reproduce steps with local ccm nodes (3 nodes): {noformat} CREATE KEYSPACE zjay_test WITH replication =3D {'class': 'SimpleStrategy', = 'replication_factor': '3'} echo 1,2,3, | parallel -vkd, "/Users/zjay/ws/cassandra/bin/cqlsh 127.0.0.{}= -e 'create table zjay_test.tt4(tt int PRIMARY KEY, name text, id text)'" {noformat} Rolling restart the cluster will fix the problem, just wondering if there's= any better way to recover from this issue. was (Author: jay.zhuang): Is there any mitigation other than rolling restart the cluster? We saw the = problem in multiple clusters (version 3.0.14), it's causing other table cre= ation returns: {noformat} ./bin/cqlsh 127.0.0.1 -e "create table zjay_test.tt81(tt int PRIMARY KEY, n= ame text, id text)" :1:ServerError: java.lang.RuntimeException: java.util.concurrent.Exe= cutionException: org.apache.cassandra.exceptions.ConfigurationException: Co= lumn family ID mismatch for keyspace zjay_test table tt4 (found 8f64bab0-a8= 9b-11e7-bca7-8d4000ea9db6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) {noformat} The table is successfully created but the schema id is never changed, I gue= ss it's because the migration task failed: {noformat} DEBUG [MigrationStage:1] 2017-10-04 11:44:43,185 Schema.java:465 - Adding o= rg.apache.cassandra.config.CFMetaData@540ff3c8[cfId=3D1605ec90-a934-11e7-b0= 3f-237cabe3739e,ksName=3Dzjay_test,cfName=3Dtt81,flags=3D[COMPOUND],params= =3DTableParams{comment=3D, read_repair_chance=3D0.0, dclocal_read_repair_ chance=3D0.1, bloom_filter_fp_chance=3D0.01, crc_check_chance=3D1.0, gc_gra= ce_seconds=3D864000, default_time_to_live=3D0, memtable_flush_period_in_ms= =3D0, min_index_interval=3D128, max_index_interval=3D2048, speculative_retr= y=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_per_partition' : 'NONE'}= , co mpaction=3DCompactionParams{class=3Dorg.apache.cassandra.db.compaction.Size= TieredCompactionStrategy, options=3D{max_threshold=3D32, min_threshold=3D4}= }, compression=3Dorg.apache.cassandra.schema.CompressionParams@78dfc45f, ex= tensions=3D{}, cdc=3Dfalse},comparator=3Dcomparator(),partitionColumns=3D[[= ] | [id name]],partitionKeyColumns=3D[ColumnDefinition{name=3Dtt, type=3Dorg.a= pache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}],= clusteringColumns=3D[],keyValidator=3Dorg.apache.cassandra.db.marshal.Int32= Type,columnMetadata=3D[ColumnDefinition{name=3Dtt, type=3Dorg.apache.cassan= dra .db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}, ColumnDefinitio= n{name=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DREGULA= R, position=3D-1}, ColumnDefinition{name=3Dname, type=3Dorg.apache.cassandr= a.db.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}],droppedColumns=3D{},= trigg ers=3D[],indexes=3D[]] to cfIdMap DEBUG [MigrationStage:1] 2017-10-04 11:44:43,186 CFMetaData.java:745 - appl= ying org.apache.cassandra.config.CFMetaData@6e7e6efb[cfId=3D8f64bab0-a89b-1= 1e7-bca7-8d4000ea9db6,ksName=3Dzjay_test,cfName=3Dtt4,flags=3D[COMPOUND],pa= rams=3DTableParams{comment=3D, read_repair_chance=3D0.0, dclocal_read_re pair_chance=3D0.1, bloom_filter_fp_chance=3D0.01, crc_check_chance=3D1.0, g= c_grace_seconds=3D864000, default_time_to_live=3D0, memtable_flush_period_i= n_ms=3D0, min_index_interval=3D128, max_index_interval=3D2048, speculative_= retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_per_partition' : 'NO= NE' }, compaction=3DCompactionParams{class=3Dorg.apache.cassandra.db.compaction= .SizeTieredCompactionStrategy, options=3D{max_threshold=3D32, min_threshold= =3D4}}, compression=3Dorg.apache.cassandra.schema.CompressionParams@78dfc45= f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcomparator(),partitionColumns= =3D [[] | [id name]],partitionKeyColumns=3D[ColumnDefinition{name=3Dtt, type=3D= org.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position= =3D0}],clusteringColumns=3D[],keyValidator=3Dorg.apache.cassandra.db.marsha= l.Int32Type,columnMetadata=3D[ColumnDefinition{name=3Dtt, type=3Dorg.apache= .cass andra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}, ColumnDefi= nition{name=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DR= EGULAR, position=3D-1}, ColumnDefinition{name=3Dname, type=3Dorg.apache.cas= sandra.db.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}],droppedColumns= =3D{}, triggers=3D[],indexes=3D[]] to org.apache.cassandra.config.CFMetaData@69e66= 630[cfId=3D8f5ccb70-a89b-11e7-b03f-237cabe3739e,ksName=3Dzjay_test,cfName= =3Dtt4,flags=3D[COMPOUND],params=3DTableParams{comment=3D, read_repair_chan= ce=3D0.0, dclocal_read_repair_chance=3D0.1, bloom_filter_fp_chance=3D0.01, = crc_che ck_chance=3D1.0, gc_grace_seconds=3D864000, default_time_to_live=3D0, memta= ble_flush_period_in_ms=3D0, min_index_interval=3D128, max_index_interval=3D= 2048, speculative_retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_pe= r_partition' : 'NONE'}, compaction=3DCompactionParams{class=3Dorg.apache.ca= ssa ndra.db.compaction.SizeTieredCompactionStrategy, options=3D{max_threshold= =3D32, min_threshold=3D4}}, compression=3Dorg.apache.cassandra.schema.Compr= essionParams@78dfc45f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcomparato= r(),partitionColumns=3D[[] | [id name]],partitionKeyColumns=3D[ColumnDefini= tio n{name=3Dtt, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTI= TION_KEY, position=3D0}],clusteringColumns=3D[],keyValidator=3Dorg.apache.c= assandra.db.marshal.Int32Type,columnMetadata=3D[ColumnDefinition{name=3Dtt,= type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, po= sit ion=3D0}, ColumnDefinition{name=3Did, type=3Dorg.apache.cassandra.db.marsha= l.UTF8Type, kind=3DREGULAR, position=3D-1}, ColumnDefinition{name=3Dname, t= ype=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DREGULAR, position=3D= -1}],droppedColumns=3D{},triggers=3D[],indexes=3D[]] ERROR [MigrationStage:1] 2017-10-04 11:44:43,187 CassandraDaemon.java:207 -= Exception in thread Thread[MigrationStage:1,5,main] org.apache.cassandra.exceptions.ConfigurationException: Column family ID mi= smatch for keyspace zjay_test table tt4 (found 8f64bab0-a89b-11e7-bca7-8d40= 00ea9db6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaD= ata.java:785) ~[main/:na] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[= main/:na] at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[ma= in/:na] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeys= pace.java:1395) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspac= e.java:1351) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVer= sion(SchemaKeyspace.java:1301) ~[main/:na] at org.apache.cassandra.service.MigrationManager$1.runMayThrow(Migratio= nManager.java:543) ~[main/:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:= 28) ~[main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) ~[na:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_= 141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) ~[na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) [na:1.8.0_141] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca= lDeallocator$0(NamedThreadFactory.java:79) [main/:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141] ERROR [SharedPool-Worker-7] 2017-10-04 11:44:43,188 QueryMessage.java:128 -= Unexpected error during query java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.ap= ache.cassandra.exceptions.ConfigurationException: Column family ID mismatch= for keyspace zjay_test table tt4 (found 8f64bab0-a89b-11e7-bca7-8d4000ea9d= b6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java= :402) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announce(MigrationMana= ger.java:525) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamil= y(MigrationManager.java:355) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamil= y(MigrationManager.java:340) ~[main/:na] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamil= y(MigrationManager.java:320) ~[main/:na] at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMi= gration(CreateTableStatement.java:87) ~[main/:na] at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute= (SchemaAlteringStatement.java:93) ~[main/:na] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProce= ssor.java:206) ~[main/:na] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java= :237) ~[main/:na] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java= :222) ~[main/:na] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMe= ssage.java:115) ~[main/:na] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Messa= ge.java:513) [main/:na] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Messa= ge.java:407) [main/:na] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChann= elInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abs= tractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Fin= al] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractCh= annelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannel= HandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) [na:1.8.0_141] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$Fu= tureTask.run(AbstractLocalAwareExecutorService.java:164) [main/:na] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [m= ain/:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141] {noformat} Schema push also failed on other nodes: {noformat} DEBUG [InternalResponseStage:19] 2017-10-04 14:52:20,324 CFMetaData.java:74= 5 - applying org.apache.cassandra.config.CFMetaData@6c71d136[cfId=3D8f64bab= 0-a89b-11e7-bca7-8d4000ea9db6,ksName=3Dzjay_test,cfName=3Dtt4,flags=3D[COMP= OUND],params=3DTableParams{comment=3D, read_repair_chance=3D0.0, dclocal _read_repair_chance=3D0.1, bloom_filter_fp_chance=3D0.01, crc_check_chance= =3D1.0, gc_grace_seconds=3D864000, default_time_to_live=3D0, memtable_flush= _period_in_ms=3D0, min_index_interval=3D128, max_index_interval=3D2048, spe= culative_retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', 'rows_per_partiti= on' : 'NONE'}, compaction=3DCompactionParams{class=3Dorg.apache.cassandra.db.co= mpaction.SizeTieredCompactionStrategy, options=3D{max_threshold=3D32, min_t= hreshold=3D4}}, compression=3Dorg.apache.cassandra.schema.CompressionParams= @78dfc45f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcomparator(),partitio= n Columns=3D[[] | [id name]],partitionKeyColumns=3D[ColumnDefinition{name=3Dt= t, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, = position=3D0}],clusteringColumns=3D[],keyValidator=3Dorg.apache.cassandra.d= b.marshal.Int32Type,columnMetadata=3D[ColumnDefinition{name=3Dtt, type=3Dor= g.apa che.cassandra.db.marshal.Int32Type, kind=3DPARTITION_KEY, position=3D0}, Co= lumnDefinition{name=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, = kind=3DREGULAR, position=3D-1}, ColumnDefinition{name=3Dname, type=3Dorg.ap= ache.cassandra.db.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}],dropped= Col umns=3D{},triggers=3D[],indexes=3D[]] to org.apache.cassandra.config.CFMeta= Data@24fcccbc[cfId=3D8f5ccb70-a89b-11e7-b03f-237cabe3739e,ksName=3Dzjay_tes= t,cfName=3Dtt4,flags=3D[COMPOUND],params=3DTableParams{comment=3D, read_rep= air_chance=3D0.0, dclocal_read_repair_chance=3D0.1, bloom_filter_fp_chance= =3D0.01, crc_check_chance=3D1.0, gc_grace_seconds=3D864000, default_time_to_live=3D= 0, memtable_flush_period_in_ms=3D0, min_index_interval=3D128, max_index_int= erval=3D2048, speculative_retry=3D99PERCENTILE, caching=3D{'keys' : 'ALL', = 'rows_per_partition' : 'NONE'}, compaction=3DCompactionParams{class=3Dorg.a= pac he.cassandra.db.compaction.SizeTieredCompactionStrategy, options=3D{max_thr= eshold=3D32, min_threshold=3D4}}, compression=3Dorg.apache.cassandra.schema= .CompressionParams@78dfc45f, extensions=3D{}, cdc=3Dfalse},comparator=3Dcom= parator(),partitionColumns=3D[[] | [id name]],partitionKeyColumns=3D[Column= D efinition{name=3Dtt, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind= =3DPARTITION_KEY, position=3D0}],clusteringColumns=3D[],keyValidator=3Dorg.= apache.cassandra.db.marshal.Int32Type,columnMetadata=3D[ColumnDefinition{na= me=3Dtt, type=3Dorg.apache.cassandra.db.marshal.Int32Type, kind=3DPARTITION= _KE Y, position=3D0}, ColumnDefinition{name=3Did, type=3Dorg.apache.cassandra.d= b.marshal.UTF8Type, kind=3DREGULAR, position=3D-1}, ColumnDefinition{name= =3Dname, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DREGULAR, p= osition=3D-1}],droppedColumns=3D{},triggers=3D[],indexes=3D[]] ERROR [InternalResponseStage:19] 2017-10-04 14:52:20,325 MigrationTask.java= :96 - Configuration exception merging remote schema org.apache.cassandra.exceptions.ConfigurationException: Column family ID mi= smatch for keyspace zjay_test table tt4 (found 8f64bab0-a89b-11e7-bca7-8d40= 00ea9db6; expected 8f5ccb70-a89b-11e7-b03f-237cabe3739e) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaD= ata.java:785) ~[main/:na] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[= main/:na] at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[ma= in/:na] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeys= pace.java:1395) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspac= e.java:1351) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVer= sion(SchemaKeyspace.java:1301) ~[main/:na] at org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.= java:92) ~[main/:na] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHand= ler.java:53) [main/:na] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask= .java:67) [main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) [na:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_1= 41] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) [na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) [na:1.8.0_141] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca= lDeallocator$0(NamedThreadFactory.java:79) [main/:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141] {noformat} I tried `nodetool resetlocalschema`, which causes following failure: {noformat} INFO [MigrationStage:1] 2017-10-04 14:50:51,677 ColumnFamilyStore.java:389= - Initializing zjay_test.tt4 ERROR [MigrationStage:1] 2017-10-04 14:50:51,679 CassandraDaemon.java:207 -= Exception in thread Thread[MigrationStage:1,5,main] java.lang.RuntimeException: javax.management.InstanceAlreadyExistsException= : org.apache.cassandra.db:type=3DTables,keyspace=3Dzjay_test,table=3Dtt4 at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.j= ava:439) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.j= ava:361) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(Co= lumnFamilyStore.java:577) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(Co= lumnFamilyStore.java:554) ~[main/:na] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:368) ~[main/:n= a] at org.apache.cassandra.config.Schema.addTable(Schema.java:649) ~[main/= :na] at java.util.HashMap$Values.forEach(HashMap.java:980) ~[na:1.8.0_141] at java.util.Collections$UnmodifiableCollection.forEach(Collections.jav= a:1080) ~[na:1.8.0_141] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeys= pace.java:1386) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspac= e.java:1351) ~[main/:na] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVer= sion(SchemaKeyspace.java:1301) ~[main/:na] at org.apache.cassandra.service.MigrationManager$1.runMayThrow(Migratio= nManager.java:543) ~[main/:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:= 28) ~[main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) ~[na:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_= 141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) ~[na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) [na:1.8.0_141] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca= lDeallocator$0(NamedThreadFactory.java:79) [main/:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141] Caused by: javax.management.InstanceAlreadyExistsException: org.apache.cass= andra.db:type=3DTables,keyspace=3Dzjay_test,table=3Dtt4 at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) ~[n= a:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRe= pository(DefaultMBeanServerInterceptor.java:1898) ~[na:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynami= cMBean(DefaultMBeanServerInterceptor.java:966) ~[na:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject= (DefaultMBeanServerInterceptor.java:900) ~[na:1.8.0_141] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(= DefaultMBeanServerInterceptor.java:324) ~[na:1.8.0_141] at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.= java:522) ~[na:1.8.0_141] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.j= ava:434) ~[main/:na] ... 18 common frames omitted {noformat} and inconsistent schema ID: {noformat} Cluster Information: Name: tt2 Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: 9a47fb8d-1712-362e-b422-6331fe448d10: [127.0.0.3] 776aa76b-1c7a-3bf8-9891-4564b72981eb: [127.0.0.1] 59adb24e-f3cd-3e02-97f0-5b395827453f: [127.0.0.2] {noformat} Here is the reproduce steps with local ccm nodes (3 nodes): {noformat} CREATE KEYSPACE zjay_test WITH replication =3D {'class': 'SimpleStrategy', = 'replication_factor': '3'} echo 1,2,3, | parallel -vkd, "/Users/zjay/ws/cassandra/bin/cqlsh 127.0.0.{}= -e 'create table zjay_test.tt4(tt int PRIMARY KEY, name text, id text)'" {noformat} Rolling restart the cluster will fix the problem, just wondering if there's= any better way to recover from this issue. > Schema inconsistency (cached vs schema_columnfamilies) > ------------------------------------------------------ > > Key: CASSANDRA-8387 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8387 > Project: Cassandra > Issue Type: Bug > Environment: C* 2.1.1 3-node cluster > Reporter: Marcus Olsson > > While running some tests on a 3-node cluster running C* 2.1.1 we encounte= red a problem creating the same table schema twice(on different nodes). One= thing to note is that one of the nodes clock was ~4 seconds behind the oth= ers, but I don't think that's the problem since the exception was reproduce= d here aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg395= 60.html. > While running the same create table statement more than once(on different= clients) the logs outputted this on one of the nodes: > {noformat} > (node x.x.x.1): > 2014-11-25T16:11:44.651+0100 INFO [SharedPool-Worker-2] MigrationManager= .java:248 Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@4= 5c290de[cfId=3D5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=3Dtest,cfName=3D= test,cfType=3DStandard,comparator=3Dorg.apache.cassandra.db.marshal.UTF8Typ= e,comment=3D,readRepairChance=3D0.0,dcLocalReadRepairChance=3D0.1,gcGraceSe= conds=3D864000,defaultValidator=3Dorg.apache.cassandra.db.marshal.BytesType= ,keyValidator=3Dorg.apache.cassandra.db.marshal.UTF8Type,minCompactionThres= hold=3D4,maxCompactionThreshold=3D32,columnMetadata=3D[ColumnDefinition{nam= e=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DCLUSTERING_= COLUMN, componentIndex=3Dnull, indexName=3Dnull, indexType=3Dnull}, ColumnD= efinition{name=3Dkey, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind= =3DPARTITION_KEY, componentIndex=3Dnull, indexName=3Dnull, indexType=3Dnull= }, ColumnDefinition{name=3Dvalue, type=3Dorg.apache.cassandra.db.marshal.By= tesType, kind=3DCOMPACT_VALUE, componentIndex=3Dnull, indexName=3Dnull, ind= exType=3Dnull}],compactionStrategyClass=3Dclass org.apache.cassandra.db.com= paction.SizeTieredCompactionStrategy,compactionStrategyOptions=3D{},compres= sionParameters=3D{sstable_compression=3Dorg.apache.cassandra.io.compress.LZ= 4Compressor},bloomFilterFpChance=3D0.01,memtableFlushPeriod=3D0,caching=3D{= "keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=3D0,minIndexIn= terval=3D128,maxIndexInterval=3D2048,speculativeRetry=3D99.0PERCENTILE,drop= pedColumns=3D{},triggers=3D[],isDense=3Dtrue] > ... > 2014-11-25T16:11:44.667+0100 INFO [MigrationStage:1] DefsTables.java:373= Loading org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=3D5bc7c980-74= b5-11e4-9131-d9b94a3d8927,ksName=3Dtest,cfName=3Dtest,cfType=3DStandard,com= parator=3Dorg.apache.cassandra.db.marshal.UTF8Type,comment=3D,readRepairCha= nce=3D0.0,dcLocalReadRepairChance=3D0.1,gcGraceSeconds=3D864000,defaultVali= dator=3Dorg.apache.cassandra.db.marshal.BytesType,keyValidator=3Dorg.apache= .cassandra.db.marshal.UTF8Type,minCompactionThreshold=3D4,maxCompactionThre= shold=3D32,columnMetadata=3D[ColumnDefinition{name=3Did, type=3Dorg.apache.= cassandra.db.marshal.UTF8Type, kind=3DCLUSTERING_COLUMN, componentIndex=3Dn= ull, indexName=3Dnull, indexType=3Dnull}, ColumnDefinition{name=3Dkey, type= =3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DPARTITION_KEY, componen= tIndex=3Dnull, indexName=3Dnull, indexType=3Dnull}, ColumnDefinition{name= =3Dvalue, type=3Dorg.apache.cassandra.db.marshal.BytesType, kind=3DCOMPACT_= VALUE, componentIndex=3Dnull, indexName=3Dnull, indexType=3Dnull}],compacti= onStrategyClass=3Dclass org.apache.cassandra.db.compaction.SizeTieredCompac= tionStrategy,compactionStrategyOptions=3D{},compressionParameters=3D{sstabl= e_compression=3Dorg.apache.cassandra.io.compress.LZ4Compressor},bloomFilter= FpChance=3D0.01,memtableFlushPeriod=3D0,caching=3D{"keys":"ALL", "rows_per_= partition":"NONE"},defaultTimeToLive=3D0,minIndexInterval=3D128,maxIndexInt= erval=3D2048,speculativeRetry=3D99.0PERCENTILE,droppedColumns=3D{},triggers= =3D[],isDense=3Dtrue] > ... > java.lang.RuntimeException: org.apache.cassandra.exceptions.Configuration= Exception: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad06= 89f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1171) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTabl= es.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTab= les.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTab= les.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java= :166) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.service.MigrationManager$2.runMayThrow(Mi= grationManager.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:= 1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) ~[na:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column= family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5= bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.validateCompatility(CFM= etaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1= 186) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1167) ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > 2014-11-25T16:11:44.859+0100 ERROR [SharedPool-Worker-2] ErrorMessage.jav= a:243 Unexpected exception during request > java.lang.RuntimeException: java.util.concurrent.ExecutionException: java= .lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationExcept= ion: Column family ID mismatch (found 5e334b40-74b5-11e4- > b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilitie= s.java:397) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.service.MigrationManager.announce(Migrati= onManager.java:374) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.service.MigrationManager.announceNewColum= nFamily(MigrationManager.java:249) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.cql3.statements.CreateTableStatement.anno= unceMigration(CreateTableStatement.java:114) ~[apache-cassandra-2.1.1.jar:2= .1.1] > at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.e= xecute(SchemaAlteringStatement.java:80) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.cql3.QueryProcessor.processStatement(Quer= yProcessor.java:226) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.cql3.QueryProcessor.processPrepared(Query= Processor.java:481) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.transport.messages.ExecuteMessage.execute= (ExecuteMessage.java:133) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.transport.Message$Dispatcher.channelRead0= (Message.java:439) [apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.transport.Message$Dispatcher.channelRead0= (Message.java:335) [apache-cassandra-2.1.1.jar:2.1.1] > at io.netty.channel.SimpleChannelInboundHandler.channelRead(Simpl= eChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRe= ad(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.= 23.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$700(Abst= ractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final= ] > at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractC= hannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) [na:1.7.0_51] > at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorSe= rvice$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-= cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:1= 05) [apache-cassandra-2.1.1.jar:2.1.1] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeExce= ption: org.apache.cassandra.exceptions.ConfigurationException: Column famil= y ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d > ; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[= na:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:= 1.7.0_51] > at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilitie= s.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 17 common frames omitted > Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.Co= nfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b= 1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1171) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTabl= es.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTab= les.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTab= les.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java= :166) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.service.MigrationManager$2.runMayThrow(Mi= grationManager.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) [na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:= 1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) ~[na:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) ~[na:1.7.0_51] > ... 1 common frames omitted > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column= family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5= bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.validateCompatility(CFM= etaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1= 186) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1167) ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > {noformat} > In my understanding this means that it first created the table(with UUID = 5e..) and then got a schema update from another node with UUID 5b.. . > And the node who created the 5b.. table had this: > {noformat} > (node x.x.x.2): > 2014-11-25T16:11:40.580+0100 INFO [SharedPool-Worker-1] MigrationManager= .java:248 Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@2= 25c1975[cfId=3D5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=3Dtest,cfName=3D= test,cfType=3DStandard,comparator=3Dorg.apache.cassandra.db.marshal.UTF8Typ= e,comment=3D,readRepairChance=3D0.0,dcLocalReadRepairChance=3D0.1,gcGraceSe= conds=3D864000,defaultValidator=3Dorg.apache.cassandra.db.marshal.BytesType= ,keyValidator=3Dorg.apache.cassandra.db.marshal.UTF8Type,minCompactionThres= hold=3D4,maxCompactionThreshold=3D32,columnMetadata=3D[ColumnDefinition{nam= e=3Did, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DCLUSTERING_= COLUMN, componentIndex=3Dnull, indexName=3Dnull, indexType=3Dnull}, ColumnD= efinition{name=3Dkey, type=3Dorg.apache.cassandra.db.marshal.UTF8Type, kind= =3DPARTITION_KEY, componentIndex=3Dnull, indexName=3Dnull, indexType=3Dnull= }, ColumnDefinition{name=3Dvalue, type=3Dorg.apache.cassandra.db.marshal.By= tesType, kind=3DCOMPACT_VALUE, componentIndex=3Dnull, indexName=3Dnull, ind= exType=3Dnull}],compactionStrategyClass=3Dclass org.apache.cassandra.db.com= paction.SizeTieredCompactionStrategy,compactionStrategyOptions=3D{},compres= sionParameters=3D{sstable_compression=3Dorg.apache.cassandra.io.compress.LZ= 4Compressor},bloomFilterFpChance=3D0.01,memtableFlushPeriod=3D0,caching=3D{= "keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=3D0,minIndexIn= terval=3D128,maxIndexInterval=3D2048,speculativeRetry=3D99.0PERCENTILE,drop= pedColumns=3D{},triggers=3D[],isDense=3Dtrue] > ... > java.lang.RuntimeException: org.apache.cassandra.exceptions.Configuration= Exception: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad06= 89f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1171) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTabl= es.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTab= les.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTab= les.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java= :166) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayT= hrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.1.jar:2.1= .1] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:= 1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) ~[na:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column= family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5= bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.validateCompatility(CFM= etaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1= 186) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1167) ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > {noformat} > And the last node had this: > {noformat} > (node x.x.x.3): > 2014-11-25T16:11:44.363+0100 INFO [MigrationStage:1] DefsTables.java:373= Loading org.apache.cassandra.config.CFMetaData@767512bf[cfId=3D5bc7c980-74= b5-11e4-9131-d9b94a3d8927,ksName=3Dtest,cfName=3Dtest,cfType=3DStandard,com= parator=3Dorg.apache.cassandra.db.marshal.UTF8Type,comment=3D,readRepairCha= nce=3D0.0,dcLocalReadRepairChance=3D0.1,gcGraceSeconds=3D864000,defaultVali= dator=3Dorg.apache.cassandra.db.marshal.BytesType,keyValidator=3Dorg.apache= .cassandra.db.marshal.UTF8Type,minCompactionThreshold=3D4,maxCompactionThre= shold=3D32,columnMetadata=3D[ColumnDefinition{name=3Did, type=3Dorg.apache.= cassandra.db.marshal.UTF8Type, kind=3DCLUSTERING_COLUMN, componentIndex=3Dn= ull, indexName=3Dnull, indexType=3Dnull}, ColumnDefinition{name=3Dkey, type= =3Dorg.apache.cassandra.db.marshal.UTF8Type, kind=3DPARTITION_KEY, componen= tIndex=3Dnull, indexName=3Dnull, indexType=3Dnull}, ColumnDefinition{name= =3Dvalue, type=3Dorg.apache.cassandra.db.marshal.BytesType, kind=3DCOMPACT_= VALUE, componentIndex=3Dnull, indexName=3Dnull, indexType=3Dnull}],compacti= onStrategyClass=3Dclass org.apache.cassandra.db.compaction.SizeTieredCompac= tionStrategy,compactionStrategyOptions=3D{},compressionParameters=3D{sstabl= e_compression=3Dorg.apache.cassandra.io.compress.LZ4Compressor},bloomFilter= FpChance=3D0.01,memtableFlushPeriod=3D0,caching=3D{"keys":"ALL", "rows_per_= partition":"NONE"},defaultTimeToLive=3D0,minIndexInterval=3D128,maxIndexInt= erval=3D2048,speculativeRetry=3D99.0PERCENTILE,droppedColumns=3D{},triggers= =3D[],isDense=3Dtrue] > ... > java.lang.RuntimeException: org.apache.cassandra.exceptions.Configuration= Exception: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad06= 89f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1171) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTabl= es.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTab= les.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTab= les.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java= :166) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayT= hrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.1.jar:2.1= .1] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:= 1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) ~[na:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column= family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5= bc7c980-74b5-11e4-9131-d9b94a3d8927) > at org.apache.cassandra.config.CFMetaData.validateCompatility(CFM= etaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1= 186) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:= 1167) ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > {noformat} > Not a problem at all, since they all seemed to agree that 5b.. was the sc= hema to go with and everything went fine with inserting data into the table= . The data directory contained the 5b.. directory and not 5e.. (on all node= s). The problem occured later when node x.x.x.3 was restarted it created th= e directory test/test-5e.. in the data directory. When performing a repair = on the node it gave these kinds of exceptions: > {noformat} > (node x.x.x.3): > 2014-11-25T16:21:31.726+0100 WARN [STREAM-IN-/x.x.x.2] StreamSession.jav= a:590 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] Retrying for following= error > java.io.IOException: CF 5bc7c980-74b5-11e4-9131-d9b94a3d8927 was dropped = during streaming > at org.apache.cassandra.streaming.compress.CompressedStreamReader= .read(CompressedStreamReader.java:71) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.= deserialize(IncomingFileMessage.java:48) [apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.= deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.messages.StreamMessage.deserial= ize(StreamMessage.java:55) [apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessa= geHandler.run(ConnectionHandler.java:245) [apache-cassandra-2.1.1.jar:2.1.1= ] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > ... > 2014-11-25T16:21:31.726+0100 ERROR [STREAM-IN-/x.x.x.2] StreamSession.jav= a:470 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] Streaming error occurr= ed > java.lang.IllegalArgumentException: Unknown type 0 > at org.apache.cassandra.streaming.messages.StreamMessage$Type.get= (StreamMessage.java:89) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.messages.StreamMessage.deserial= ize(StreamMessage.java:54) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessa= geHandler.run(ConnectionHandler.java:245) ~[apache-cassandra-2.1.1.jar:2.1.= 1] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > ... > 2014-11-25T16:21:31.738+0100 ERROR [AntiEntropySessions:1] RepairSession.= java:303 [repair #bbf34630-74b6-11e4-baf6-fbb25862655d] session completed w= ith the following error > org.apache.cassandra.exceptions.RepairException: [repair #bbf34630-74b6-1= 1e4-baf6-fbb25862655d on test/test, (3708140006327119811,372136091680044661= 1]] Sync failed between /x.x.x.3 and > /x.x.x.2 > at org.apache.cassandra.repair.RepairSession.syncComplete(RepairS= ession.java:223) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.service.ActiveRepairService.handleMessage= (ActiveRepairService.java:389) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(Re= pairMessageVerbHandler.java:126) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDelive= ryTask.java:62) ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) [na:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > {noformat} > And the other nodes had exceptions like: > {noformat} > (node x.x.x.2) > 2014-11-25T16:21:28.145+0100 ERROR [STREAM-OUT-/x.x.x.3] StreamSession.ja= va:470 [Stream #bc309d50-74b6-11e4-baf6-fbb25862655d] Streaming error occur= red > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0= _51] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[= na:1.7.0_51] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1= .7.0_51] > at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)= ~[na:1.7.0_51] > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[= na:1.7.0_51] > at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1= .7.0_51] > at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1= .7.0_51] > at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.7= .0_51] > at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.7.0_51= ] > at java.nio.channels.Channels$1.write(Channels.java:155) ~[na:1.7= .0_51] > at org.apache.cassandra.io.util.DataOutputStreamPlus.write(DataOu= tputStreamPlus.java:45) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.io.util.AbstractDataOutput.writeLong(Abst= ractDataOutput.java:232) ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSeri= alizer.serialize(CompressionMetadata.java:406) ~[apache-cassandra-2.1.1.jar= :2.1.1] > at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSeri= alizer.serialize(CompressionMetadata.java:402) ~[apache-cassandra-2.1.1.jar= :2.1.1] > at org.apache.cassandra.streaming.compress.CompressionInfo$Compre= ssionInfoSerializer.serialize(CompressionInfo.java:60) ~[apache-cassandra-2= .1.1.jar:2.1.1] > at org.apache.cassandra.streaming.compress.CompressionInfo$Compre= ssionInfoSerializer.serialize(CompressionInfo.java:47) ~[apache-cassandra-2= .1.1.jar:2.1.1] > at org.apache.cassandra.streaming.messages.FileMessageHeader$File= MessageHeaderSerializer.serialize(FileMessageHeader.java:135) ~[apache-cass= andra-2.1.1.jar:2.1.1] > at org.apache.cassandra.streaming.messages.FileMessageHeader$File= MessageHeaderSerializer.serialize(FileMessageHeader.java:120) ~[apache-cass= andra-2.1.1.jar:2.1.1] > {noformat} > In System.schema_columnfamilies the id for test.test is 5e.. on all nodes= , but the nodes that hadn't restarted still used the 5b.. id for the table = (atleast in the data directory, probably a cached id). > The problem seems to be that the schema change for 5e.. is rejected on al= l nodes (java.lang.RuntimeException: org.apache.cassandra.exceptions.Config= urationException thrown), while the mutation for it still gets applied in s= chema_columnfamilies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org