incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikael Wikblom <mikael.wikb...@sitevision.se>
Subject org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
Date Tue, 01 Feb 2011 13:27:50 GMT
Hi,

I'm getting an exception when doing many writes to a column with an
empty column name (i.e. empty byte buffer). The exception occurs during
compactation. It is reproducable given enough writes (around 1300000).
Is this a known issue? After the problem occurs cassandra will no longer
start up. It starts up again after I've manually deleted the commit log
files.

I have failed to reproduce the problem with a non empty byte array.

I'm using cassandra 0.7

Is there a way to recover from such errors without loosing the entries
in the commit log?

Regards

Mikael Wikblom



Here is my unit test, the exception and the yaml. I also provided the
stacktrace from the following startup exception

===unit test==============================

@Test
   public void testInvalidColumnNames()
         throws Exception
   {
      final Charset CHARSET = Charset.forName("UTF-8");

      int count = 1300000;
      for (int i = 0; i < count; i++)
      {
         List<RowMutation> mutations = new ArrayList<RowMutation>();
         RowMutation m = new RowMutation("Test",
ByteBuffer.wrap(("rowId").getBytes(CHARSET)));
         // QueryPath qp = new QueryPath("TestCassandra", null,
ByteBuffer.wrap(("test" + i).getBytes(CHARSET)));
         QueryPath qp = new QueryPath("TestCassandra", null,
FBUtilities.EMPTY_BYTE_BUFFER);

         m.add(qp, ByteBuffer.wrap("test".getBytes(CHARSET)),
System.currentTimeMillis());
         mutations.add(m);

         StorageProxy.mutate(mutations, ConsistencyLevel.ONE);
      }

      StorageService.instance.forceTableFlush(CassandraInitializer.KS,
"TestCassandra");

      Thread.sleep(30*1000);
   }


===exception when running test=========================================

13:54:38 ERROR [PrecompactedRow.<init>]  Skipping row
DecoratedKey(rowId, 726f774964) in
./target/test/cassandra/data/Test/TestCassandra-e-4-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
    at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
    at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
    at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
    at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
    at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
    at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:137)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:132)
    at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
    at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
    at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:312)
    at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
    at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

===yaml file============================================

# Cassandra YAML generated from previous config
# Configuration wiki: http://wiki.apache.org/cassandra/StorageConfiguration
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
auto_bootstrap: false
binary_memtable_throughput_in_mb: 256
cluster_name: CluperUnitTest
column_index_size_in_kb: 64
commitlog_directory: ./target/test/cassandra/commitlog
commitlog_rotation_threshold_in_mb: 128
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
compaction_thread_priority: 1
concurrent_reads: 8
concurrent_writes: 32
data_file_directories:
- ./target/test/cassandra/data
disk_access_mode: auto
dynamic_snitch: false
dynamic_snitch_badness_threshold: 0.0
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_update_interval_in_ms: 100
endpoint_snitch: org.apache.cassandra.locator.RackInferringSnitch
hinted_handoff_enabled: true
in_memory_compaction_limit_in_mb: 256
index_interval: 128
saved_caches_directory: ./target/test/cassandra/saved_caches
keyspaces:
- column_families:
  - column_metadata: []
    column_type: Standard
    compare_with: UTF8Type
    gc_grace_seconds: 864000
    key_cache_save_period_in_seconds: 3600
    keys_cached: 200000.0
    max_compaction_threshold: 32
    memtable_flush_after_mins: 60
    memtable_operations_in_millions: 0.3
    memtable_throughput_in_mb: 64
    min_compaction_threshold: 4
    name: TestCassandra
    read_repair_chance: 1.0
    row_cache_save_period_in_seconds: 0
    rows_cached: 0.0
  name: Test
  replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
  replication_factor: 1
listen_address: localhost
partitioner: org.apache.cassandra.dht.OrderPreservingPartitioner
phi_convict_threshold: 8
rpc_keepalive: true
rpc_port: 9161
rpc_timeout_in_ms: 10000
seeds:
- 127.0.0.1
sliced_buffer_size_in_kb: 64
snapshot_before_compaction: false
storage_port: 7012
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16

===start up exception after unit test===================================

14:16:14 ERROR [AbstractCassandraDaemon.activate]  Exception encountered
during startup.
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
    at
org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:385)
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
    at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:395)
    at
org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:385)
    at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:353)
    at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:395)
    at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:283)
    at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:353)
    at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:194)
    at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:283)
    at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:143)
    at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:194)
    at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
    at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:143)
    at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217)
    at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
    at
senselogic.cluper.cassandra.CassandraInitializer$1.run(CassandraInitializer.java:67)
    at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217)
    at java.lang.Thread.run(Thread.java:662)
    at
senselogic.cluper.cassandra.CassandraInitializer$1.run(CassandraInitializer.java:67)
Exception encountered during startup.
    at java.lang.Thread.run(Thread.java:662)'

===ls of the commit log directory after the error=================

mikael@mikael2:~/projects/sitevision/trunk/framework/sitevision-framework-core/target/test-classes/target/test/cassandra/commitlog>
ls -ltr
total 187936
-rw-r--r-- 1 mikael wheel 134217730 2011-02-01 14:13
CommitLog-1296565996491.log
-rw-r--r-- 1 mikael wheel        28 2011-02-01 14:13
CommitLog-1296566017932.log.header
-rw-r--r-- 1 mikael wheel        52 2011-02-01 14:13
CommitLog-1296565996491.log.header
-rw-r--r-- 1 mikael wheel  58211952 2011-02-01 14:13
CommitLog-1296566017932.log
-rw-r--r-- 1 mikael wheel         0 2011-02-01 14:16
CommitLog-1296566174499.log
-rw-r--r-- 1 mikael wheel        20 2011-02-01 14:16
CommitLog-1296566174499.log.header

-- 
Mikael Wikblom
Software Architect
SiteVision AB
019-217058
mikael.wikblom@sitevision.se
http://www.sitevision.se


Mime
View raw message