cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryce Godfrey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4323) Joining new node to cluster fails with error in add column family
Date Sun, 19 Aug 2012 04:32:38 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437454#comment-13437454
] 

Bryce Godfrey commented on CASSANDRA-4323:
------------------------------------------

What if I'm having this issue currently and I'm on 1.1.0 ?  Do you think an upgrade to 1.1.3/4
will fix this and allow me to join new nodes again?  What got me into this situation is that
we pointed Opscenter at this cluster and it just went nuts after that.  Some of the nodes
were spamming errors about KS Opscenter does not exist, and killed my cluster.  I removed
opscenter and its agents, tried to do a drop keyspace on Opscenter (which is says does not
exist), but now I'm here without an idea of how to fix the schema corruption.
I will try upgrading to latest though and keep my fingers crossed.
                
> Joining new node to cluster fails with error in add column family
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-4323
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4323
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.1
>         Environment: CentOS 6, Java 1.6
>            Reporter: Bryce Godfrey
>
> I tried joining a new node to the cluster, and before boostrap started it reported this
error:
> INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0 serialized/live
bytes, 1 ops)
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0 serialized/live bytes,
1 ops)
> INFO 08:20:51,589 Completed flushing /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db
(61 bytes)
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1015
>         at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
>         at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
>         at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>         at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>         at org.apache.cassandra.config.Schema.load(Schema.java:385)
>         at org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>         at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
>         at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>         at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
>         at org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> INFO 08:20:51,931 Enqueuing flush of Memtable-schema_keyspaces@833041663(943/1178 serialized/live
bytes, 20 ops)
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178 serialized/live
bytes, 20 ops)
> And continued on, then started writing these errors non-stop:
> ERROR 08:21:45,959 Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1019
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>         at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>         at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>         at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>         at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> ERROR 08:21:45,814 Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1019
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>         at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>         at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>         at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>         at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> ERROR 08:21:45,813 Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1020
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>         at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>         at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>         at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>         at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> ERROR 08:21:45,813 Error in row mutation
> I had a problem a while ago of someone trying to create a new column family while a node
was hung.  The new node never picked up the new column family, so we deleted it and tried
again and everything was fine.  Not sure if its related though.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message