incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra won't restart : 7365....6c73 is not defined as a collection
Date Mon, 06 May 2013 08:14:04 GMT
Do you have the table definitions ? 
Any example data?
Something is confused about a set / map / list type. 

It's failing when replying the log, if you want to work around move the commit log file out
of the directory. There is a chance of data loss if this row mutation is being replied on
all nodes. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 3/05/2013, at 2:36 PM, Blair Zajac <blair@orcaware.com> wrote:

> Hello,
> 
> I'm running a 3-node development cluster on OpenStack VMs and recently updated to DataStax's
1.2.4 debs on Ubuntu Raring after which the cluster was fine.  I shut it down for a few days
and after getting back to Cassandra today and booting the VMs, Cassandra is unable to start.
Below is the output from output.log from one of the nodes.  None of the Cassandra nodes can
start.
> 
> The deployment is pretty simple, two test keyspaces with a few column families in each
keyspace.  I am doing a lot of keyspace and column family deletions as I'm testing some db
style migration code to auto-setup a schema.
> 
> Any suggestions?
> 
> Blair
> 
> INFO 19:24:09,780 Logging initialized
> INFO 19:24:09,790 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_21
> INFO 19:24:09,791 Heap size: 880803840/880803840
> INFO 19:24:09,791 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/netty-3.5.9.Final.jar:/usr/share/cassand!
> ra/lib/ser
> vlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.4.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.4.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
> INFO 19:24:09,987 JNA mlockall successful
> INFO 19:24:10,001 Loading settings from file:/etc/cassandra/cassandra.yaml
> INFO 19:24:10,371 Data files directories: [/var/lib/cassandra/data]
> INFO 19:24:10,372 Commit log directory: /var/lib/cassandra/commitlog
> INFO 19:24:10,372 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> INFO 19:24:10,372 disk_failure_policy is stop
> INFO 19:24:10,377 Global memtable threshold is enabled at 280MB
> INFO 19:24:10,474 Not using multi-threaded compaction
> INFO 19:24:10,816 Initializing key cache with capacity of 42 MBs.
> INFO 19:24:10,822 Scheduling key cache save to each 14400 seconds (going to save all
keys).
> INFO 19:24:10,823 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
> INFO 19:24:10,827 Scheduling row cache save to each 0 seconds (going to save all keys).
> INFO 19:24:10,958 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-165
(35 bytes)
> INFO 19:24:10,989 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-166
(168 bytes)
> INFO 19:24:10,991 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-164
(346 bytes)
> INFO 19:24:10,999 reading saved cache /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
> INFO 19:24:11,018 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-461
(6562 bytes)
> INFO 19:24:11,024 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
> INFO 19:24:11,031 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-394
(465 bytes)
> INFO 19:24:11,032 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-395
(244 bytes)
> INFO 19:24:11,033 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-393
(3025 bytes)
> INFO 19:24:11,035 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columns-KeyCache-b.db
> INFO 19:24:11,048 Opening /var/lib/cassandra/data/system/peers/system-peers-ib-16 (295
bytes)
> INFO 19:24:11,052 Opening /var/lib/cassandra/data/system/peers/system-peers-ib-15 (309
bytes)
> INFO 19:24:11,057 Opening /var/lib/cassandra/data/system/peers/system-peers-ib-14 (10840
bytes)
> INFO 19:24:11,074 Opening /var/lib/cassandra/data/system/local/system-local-ib-58 (97
bytes)
> INFO 19:24:11,076 Opening /var/lib/cassandra/data/system/local/system-local-ib-57 (97
bytes)
> INFO 19:24:11,082 Opening /var/lib/cassandra/data/system/local/system-local-ib-61 (150
bytes)
> INFO 19:24:11,086 Opening /var/lib/cassandra/data/system/local/system-local-ib-56 (148
bytes)
> INFO 19:24:11,087 Opening /var/lib/cassandra/data/system/local/system-local-ib-60 (97
bytes)
> INFO 19:24:11,089 Opening /var/lib/cassandra/data/system/local/system-local-ib-53 (5846
bytes)
> INFO 19:24:11,091 Opening /var/lib/cassandra/data/system/local/system-local-ib-55 (5410
bytes)
> INFO 19:24:11,094 Opening /var/lib/cassandra/data/system/local/system-local-ib-59 (97
bytes)
> INFO 19:24:11,095 Opening /var/lib/cassandra/data/system/local/system-local-ib-54 (120
bytes)
> INFO 19:24:11,097 reading saved cache /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
> INFO 19:24:11,666 Opening /var/lib/cassandra/data/system_auth/users/system_auth-users-ib-1
(64 bytes)
> INFO 19:24:11,708 completed pre-loading (17 keys) key cache.
> INFO 19:24:11,732 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867617.log, /var/lib/cassandra/commitlog/CommitLog-2-1367034867618.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867619.log, /var/lib/cassandra/commitlog/CommitLog-2-1367034867620.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867621.log, /var/lib/cassandra/commitlog/CommitLog-2-1367034867622.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867623.log, /var/lib/cassandra/commitlog/CommitLog-2-1367034867624.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867625.log, /var/lib/cassandra/commitlog/CommitLog-2-1367034867626.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867627.log, /var/lib/cassandra/commitlog/CommitLog-2-1367547324162.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367547728748.log, /var/lib/cassandra/commitlog/CommitLog-2-1367547828579.log
> INFO 19:24:11,783 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log
> INFO 19:24:11,870 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log
> java.lang.RuntimeException: 7365636f6e646172795f656d61696c73 is not defined as a collection
> 	at org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69)
> 	at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81)
> 	at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
> 	at java.util.TreeMap.put(TreeMap.java:545)
> 	at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
> 	at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
> 	at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
> 	at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:104)
> 	at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:376)
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:203)
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:98)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:279)
> 	at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:356)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
> Cannot load daemon


Mime
View raw message