cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Lalevée (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7188) Wrong class type: class org.apache.cassandra.db.Column in CounterColumn.reconcile
Date Wed, 01 Oct 2014 11:59:37 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154706#comment-14154706
] 

Nicolas Lalevée commented on CASSANDRA-7188:
--------------------------------------------

Our cluster was reverted back to 1.2.11. We upgraded last week to 1.2.19. Successfully, without
issues.

We tried to upgrade then to 2.0.10. Still a mess with the same error spreading around as soon
as we upgrade one node among 6.

Here is the king of errors seen on the upgrade node:
{noformat}
ERROR [ReplicateOnWriteStage:136] 2014-10-01 11:19:37,778 CassandraDaemon.java (line 199)
Exception in thread Thread[ReplicateOnWriteSt
age:136,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
        at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
        at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:109)
        at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:103)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:98)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
        at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
        at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
        at org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1086)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1928)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [MutationStage:136] 2014-10-01 11:19:36,843 CassandraDaemon.java (line 199) Exception
in thread Thread[MutationStage:136,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
        at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
        at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:326)
        at org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:191)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
        at org.apache.cassandra.db.Memtable.put(Memtable.java:165)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:900)
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:374)
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:339)
        at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
        at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

And here is what can be found on the not upgraded ones:
{noformat}
ERROR [MutationStage:147] 2014-10-01 11:26:57,270 CassandraDaemon.java (line 191) Exception
in thread Thread[MutationStage:147,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
        at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
        at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:383)
        at org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
        at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
        at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
        at org.apache.cassandra.db.ThreadSafeSortedColumns.addColumnInternal(ThreadSafeSortedColumns.java:108)
        at org.apache.cassandra.db.ThreadSafeSortedColumns.addAllWithSizeDelta(ThreadSafeSortedColumns.java:138)
        at org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:205)
        at org.apache.cassandra.db.Memtable.put(Memtable.java:168)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:749)
        at org.apache.cassandra.db.Table.apply(Table.java:389)
        at org.apache.cassandra.db.Table.apply(Table.java:354)
        at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:283)
        at org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:138)
        at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:771)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1622)
        at org.apache.cassandra.service.StorageProxy$3.apply(StorageProxy.java:141)
        at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:384)
        at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:746)
        at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:53)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

ERROR [ReadStage:73] 2014-10-01 11:27:03,218 CassandraDaemon.java (line 191) Exception in
thread Thread[ReadStage:73,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
        at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
        at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:383)
        at org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
        at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
        at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
        at org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:164)
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:141)
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
        at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:112)
        at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:96)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:111)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:103)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1398)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1130)
        at org.apache.cassandra.db.Table.getRow(Table.java:348)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

> Wrong class type: class org.apache.cassandra.db.Column in CounterColumn.reconcile
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7188
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7188
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Nicolas Lalevée
>            Assignee: Michael Shuler
>
> When migrating a cluster of 6 nodes from 1.2.11 to 2.0.7, we started to see on the first
migrated node this error:
> {noformat}
> ERROR [ReplicateOnWriteStage:1] 2014-05-07 11:26:59,779 CassandraDaemon.java (line 198)
Exception in thread Thread[ReplicateOnWriteStage:1,5,main]
> java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
>         at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
>         at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:109)
>         at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:103)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:98)
>         at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1540)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1369)
>         at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
>         at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
>         at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
>         at org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1085)
>         at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1916)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We then saw on the other 5 nodes, still on 1.2.x, this error:
> {noformat}
> ERROR [MutationStage:2793] 2014-05-07 11:46:12,301 CassandraDaemon.java (line 191) Exception
in thread Thread[MutationStage:2793,5,main]
> java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
>         at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
>         at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:378)
>         at org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
>         at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
>         at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
>         at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
>         at org.apache.cassandra.db.ThreadSafeSortedColumns.addColumnInternal(ThreadSafeSortedColumns.java:108)
>         at org.apache.cassandra.db.ThreadSafeSortedColumns.addAllWithSizeDelta(ThreadSafeSortedColumns.java:138)
>         at org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
>         at org.apache.cassandra.db.Memtable.resolve(Memtable.java:205)
>         at org.apache.cassandra.db.Memtable.put(Memtable.java:168)
>         at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:742)
>         at org.apache.cassandra.db.Table.apply(Table.java:388)
>         at org.apache.cassandra.db.Table.apply(Table.java:353)
>         at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:280)
>         at org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:137)
>         at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:773)
>         at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1651)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> {noformat}
> Here some other stack we also on the 5 unmigrated nodes:
> {noformat}
> ERROR [ReadStage:4242] 2014-05-07 11:46:12,259 CassandraDaemon.java (line 191) Exception
in thread Thread[ReadStage:4242,5,main]
> java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
>         at org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
>         at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:378)
>         at org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
>         at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
>         at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
>         at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
>         at org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:164)
>         at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:141)
>         at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
>         at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
>         at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:112)
>         at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:96)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:111)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:103)
>         at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>         at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
>         at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> {noformat}
> And the client side, it is failing with:
> {noformat}
> Caused by: org.apache.cassandra.thrift.UnavailableException: null
>         at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7866)
>         at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>         at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:594)
>         at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:578)
>         at me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:274)
> {noformat}
> After seeing such errors, we just shut down the first migrated node, hoping it would
avoid all these client errors. But errors continue to be logged, even if there were only the
5 1.2.x nodes in the ring.
> As the usual wild guess, let's reboot a node to fix it. At our damned surprise, it would
restart and would fail with:
> {noformat}
>  INFO 11:33:40,190 Initializing system.LocationInfo
> java.lang.AssertionError
>         at org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:162)
>         at org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1541)
>         at org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1456)
>         at org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:306)
>         at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:287)
>         at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:154)
>         at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:574)
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:253)
>         at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
> Cannot load daemon
> Service exit with a return value of 3
> {noformat}
> From there we only had 4 running nodes, with errors spreading around. So we halted everything,
put the first node back to 1.2.11 and restored the data which has been snapshot just before
the first node was migrated. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message