incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: AssertionError: originally calculated column size ...
Date Mon, 30 Apr 2012 04:05:14 GMT
Looks a bit like https://issues.apache.org/jira/browse/CASSANDRA-3579 but that was fixed in
1.0.7

Is this still an issue ? Are you able to reproduce the fault ? 

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 27/04/2012, at 6:56 PM, Patrik Modesto wrote:

> Hi,
> 
> I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace
> with RF=3 and one CF with two secondary indexes. I'm importing data
> into this CF using Hadoop Mapreduce job, each row has less than 10
> colkumns. From JMX:
> MaxRowSize:  1597
> MeanRowSize: 369
> 
> And there are some tens of millions of rows.
> 
> It's write-heavy usage and there is a big pressure on each node, there
> are quite some dropped mutations on each node. After ~12 hours of
> inserting I see these assertion exceptiona on 3 out of four nodes:
> 
> ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.AssertionError: originally calculated column size of
> 629444349 but now it is 588008950
>        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388)
>        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
>        at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
>        at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.AssertionError: originally calculated column size of
> 629444349 but now it is 588008950
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384)
>        ... 7 more
> Caused by: java.lang.AssertionError: originally calculated column size
> of 629444349 but now it is 588008950
>        at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124)
>        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
>        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161)
>        at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        ... 3 more
> 
> 
> Few lines regarding Hints from the output.log:
> 
> INFO 06:21:26,202 Compacting large row
> system/HintsColumnFamily:70000000000000000000000000000000 (1712834057
> bytes) incrementally
> INFO 06:22:52,610 Compacting large row
> system/HintsColumnFamily:10000000000000000000000000000000 (2616073981
> bytes) incrementally
> INFO 06:22:59,111 flushing high-traffic column family
> CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated
> 305147360 bytes)
> INFO 06:22:59,813 Enqueuing flush of
> Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live
> bytes, 7452 ops)
> INFO 06:22:59,814 Writing
> Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live
> bytes, 7452 ops)


Mime
View raw message