incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Blair Zajac <bl...@orcaware.com>
Subject Re: Cassandra won't restart : 7365....6c73 is not defined as a collection
Date Mon, 06 May 2013 17:03:38 GMT
Hi Aaron,

The keyspace consistent of 3 column families for user management, see below.

I have dropped these tables multiple times since I'm testing a script to 
automatically create the column families if they do not exists.  I have 
also been changing types, e.g. lock_tokens__ from MAP<UUID, BIGINT> to 
MAP<TEXT, BIGINT>.

I have tar copies of /var/lib/cassandra from all three nodes if somebody 
wants to look.  Since making the tarballs, I blew the cluster away and 
re-initialized it from scratch.

BTW, would a drain before running '/etc/init.d/cassandra stop' have helped?

Regards,
Blair


CREATE TABLE account (
   pk_account UUID PRIMARY KEY,
   last_login_using TEXT,
   first_name TEXT,
   last_name TEXT,
   full_name TEXT,
   created_micros BIGINT,
   modified_micros BIGINT,
   lock_tokens__ MAP<TEXT, BIGINT>
);


CREATE TABLE external_account (
   pk_external_username TEXT PRIMARY KEY,
   pk_account UUID,
   primary_email_address TEXT,
   secondary_email_addresses SET<TEXT>,
   first_name TEXT,
   last_name TEXT,
   full_name TEXT,
   last_login_micros BIGINT,
   created_micros BIGINT,
   modified_micros BIGINT,
   lock_tokens__ MAP<TEXT, BIGINT>
);

CREATE TABLE email_address (
   pk_email_address TEXT PRIMARY KEY,
   pk_account UUID,
   pk_external_username SET<TEXT>,
   lock_tokens__ MAP<TEXT, BIGINT>
);


On 05/06/2013 01:14 AM, aaron morton wrote:
> Do you have the table definitions ?
> Any example data?
> Something is confused about a set / map / list type.
>
> It's failing when replying the log, if you want to work around move the
> commit log file out of the directory. There is a chance of data loss if
> this row mutation is being replied on all nodes.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/05/2013, at 2:36 PM, Blair Zajac <blair@orcaware.com
> <mailto:blair@orcaware.com>> wrote:
>
>> Hello,
>>
>> I'm running a 3-node development cluster on OpenStack VMs and recently
>> updated to DataStax's 1.2.4 debs on Ubuntu Raring after which the
>> cluster was fine.  I shut it down for a few days and after getting
>> back to Cassandra today and booting the VMs, Cassandra is unable to
>> start. Below is the output from output.log from one of the nodes.
>>  None of the Cassandra nodes can start.
>>
>> The deployment is pretty simple, two test keyspaces with a few column
>> families in each keyspace.  I am doing a lot of keyspace and column
>> family deletions as I'm testing some db style migration code to
>> auto-setup a schema.
>>
>> Any suggestions?
>>
>> Blair
>>
>> INFO 19:24:09,780 Logging initialized
>> INFO 19:24:09,790 JVM vendor/version: Java HotSpot(TM) 64-Bit Server
>> VM/1.7.0_21
>> INFO 19:24:09,791 Heap size: 880803840/880803840
>> INFO 19:24:09,791 Classpath:
>> /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/netty-3.5.9.Final.jar:/usr/share/cass!
 and!
>> ra/lib/ser
>> vlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.4.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.4.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
>> INFO 19:24:09,987 JNA mlockall successful
>> INFO 19:24:10,001 Loading settings from file:/etc/cassandra/cassandra.yaml
>> INFO 19:24:10,371 Data files directories: [/var/lib/cassandra/data]
>> INFO 19:24:10,372 Commit log directory: /var/lib/cassandra/commitlog
>> INFO 19:24:10,372 DiskAccessMode 'auto' determined to be mmap,
>> indexAccessMode is mmap
>> INFO 19:24:10,372 disk_failure_policy is stop
>> INFO 19:24:10,377 Global memtable threshold is enabled at 280MB
>> INFO 19:24:10,474 Not using multi-threaded compaction
>> INFO 19:24:10,816 Initializing key cache with capacity of 42 MBs.
>> INFO 19:24:10,822 Scheduling key cache save to each 14400 seconds
>> (going to save all keys).
>> INFO 19:24:10,823 Initializing row cache with capacity of 0 MBs and
>> provider org.apache.cassandra.cache.SerializingCacheProvider
>> INFO 19:24:10,827 Scheduling row cache save to each 0 seconds (going
>> to save all keys).
>> INFO 19:24:10,958 Opening
>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-165
>> (35 bytes)
>> INFO 19:24:10,989 Opening
>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-166
>> (168 bytes)
>> INFO 19:24:10,991 Opening
>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-164
>> (346 bytes)
>> INFO 19:24:10,999 reading saved cache
>> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
>> INFO 19:24:11,018 Opening
>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-461
>> (6562 bytes)
>> INFO 19:24:11,024 reading saved cache
>> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
>> INFO 19:24:11,031 Opening
>> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-394
>> (465 bytes)
>> INFO 19:24:11,032 Opening
>> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-395
>> (244 bytes)
>> INFO 19:24:11,033 Opening
>> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-393
>> (3025 bytes)
>> INFO 19:24:11,035 reading saved cache
>> /var/lib/cassandra/saved_caches/system-schema_columns-KeyCache-b.db
>> INFO 19:24:11,048 Opening
>> /var/lib/cassandra/data/system/peers/system-peers-ib-16 (295 bytes)
>> INFO 19:24:11,052 Opening
>> /var/lib/cassandra/data/system/peers/system-peers-ib-15 (309 bytes)
>> INFO 19:24:11,057 Opening
>> /var/lib/cassandra/data/system/peers/system-peers-ib-14 (10840 bytes)
>> INFO 19:24:11,074 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-58 (97 bytes)
>> INFO 19:24:11,076 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-57 (97 bytes)
>> INFO 19:24:11,082 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-61 (150 bytes)
>> INFO 19:24:11,086 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-56 (148 bytes)
>> INFO 19:24:11,087 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-60 (97 bytes)
>> INFO 19:24:11,089 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-53 (5846 bytes)
>> INFO 19:24:11,091 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-55 (5410 bytes)
>> INFO 19:24:11,094 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-59 (97 bytes)
>> INFO 19:24:11,095 Opening
>> /var/lib/cassandra/data/system/local/system-local-ib-54 (120 bytes)
>> INFO 19:24:11,097 reading saved cache
>> /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
>> INFO 19:24:11,666 Opening
>> /var/lib/cassandra/data/system_auth/users/system_auth-users-ib-1 (64
>> bytes)
>> INFO 19:24:11,708 completed pre-loading (17 keys) key cache.
>> INFO 19:24:11,732 Replaying
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867617.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867618.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867619.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867620.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867621.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867622.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867623.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867624.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867625.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867626.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867627.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367547324162.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367547728748.log,
>> /var/lib/cassandra/commitlog/CommitLog-2-1367547828579.log
>> INFO 19:24:11,783 Replaying
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log
>> INFO 19:24:11,870 Finished reading
>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log
>> java.lang.RuntimeException: 7365636f6e646172795f656d61696c73 is not
>> defined as a collection
>> at
>> org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69)
>> at
>> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81)
>> at
>> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
>> at java.util.TreeMap.put(TreeMap.java:545)
>> at
>> org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
>> at
>> org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
>> at
>> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
>> at
>> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:104)
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
>> at
>> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:376)
>> at
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:203)
>> at
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:98)
>> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
>> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:279)
>> at
>> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:356)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:601)
>> at
>> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
>> Cannot load daemon
>


Mime
View raw message