incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrik Modesto <patrik.mode...@gmail.com>
Subject AssertionError: originally calculated column size ...
Date Fri, 27 Apr 2012 06:56:57 GMT
Hi,

I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace
with RF=3 and one CF with two secondary indexes. I'm importing data
into this CF using Hadoop Mapreduce job, each row has less than 10
colkumns. From JMX:
MaxRowSize:  1597
MeanRowSize: 369

And there are some tens of millions of rows.

It's write-heavy usage and there is a big pressure on each node, there
are quite some dropped mutations on each node. After ~12 hours of
inserting I see these assertion exceptiona on 3 out of four nodes:

ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of
629444349 but now it is 588008950
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
        at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
        at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of
629444349 but now it is 588008950
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384)
        ... 7 more
Caused by: java.lang.AssertionError: originally calculated column size
of 629444349 but now it is 588008950
        at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161)
        at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        ... 3 more


Few lines regarding Hints from the output.log:

 INFO 06:21:26,202 Compacting large row
system/HintsColumnFamily:70000000000000000000000000000000 (1712834057
bytes) incrementally
 INFO 06:22:52,610 Compacting large row
system/HintsColumnFamily:10000000000000000000000000000000 (2616073981
bytes) incrementally
 INFO 06:22:59,111 flushing high-traffic column family
CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated
305147360 bytes)
 INFO 06:22:59,813 Enqueuing flush of
Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live
bytes, 7452 ops)
 INFO 06:22:59,814 Writing
Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live
bytes, 7452 ops)

Mime
View raw message