The schema for the user defined CF's should not matter. The exceptions has to do with compacting the HintedHandoff CF. 

With regard to the failed read, is this happening at CL QUOURM ? 

Can you remember if the assertion was raised while the cluster was still under load ?

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 1/05/2012, at 7:11 AM, Patrik Modesto wrote:

Hi Aaron,

it's still an issue. However it's difficult to reproduce because I'd
need to insert data to Cassandra for day or more which is not
possible.

I think the problem is somehow connected to an IntegerType secondary
index. I had a different problem with CF with two secondary indexes,
the first UTF8Type, the second IntegerType. After a few hours of
inserting data in the afternoon and midnight repair+compact, the next
day I couldn't find any row using the IntegerType secondary index. The
output was like this:

[default@rfTest3] get IndexTest where col1 =
'3230727:http://zaskolak.cz/download.php';
-------------------
RowKey: 3230727:8383582:http://zaskolak.cz/download.php
=> (column=col1, value=3230727:http://zaskolak.cz/download.php,
timestamp=1335348630332000)
=> (column=col2, value=8383582, timestamp=1335348630332000)
-------------------
RowKey: 3230727:8383583:http://zaskolak.cz/download.php
=> (column=col1, value=3230727:http://zaskolak.cz/download.php,
timestamp=1335348449078000)
=> (column=col2, value=8383583, timestamp=1335348449078000)
-------------------
RowKey: 3230727:8383579:http://zaskolak.cz/download.php
=> (column=col1, value=3230727:http://zaskolak.cz/download.php,
timestamp=1335348778577000)
=> (column=col2, value=8383579, timestamp=1335348778577000)

3 Rows Returned.
Elapsed time: 292 msec(s).

[default@rfTest3] get IndexTest where col2 = 8383583;

0 Row Returned.
Elapsed time: 7 msec(s

You can see there really is an 8383583 in col2 in on of the listed
rows, but the search by secondary index returns nothing.

The Assert Exception also happend only on CF with the secondary index
of IntegerType. There were also secondary indexes of UTF8Type and
LongType types. It's the first time I've tried secondary indexes of
other type than UTF8Type.

Regards,
Patrik



On Mon, Apr 30, 2012 at 06:05, aaron morton <aaron@thelastpickle.com> wrote:
Looks a bit like https://issues.apache.org/jira/browse/CASSANDRA-3579 but
that was fixed in 1.0.7

Is this still an issue ? Are you able to reproduce the fault ?

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 27/04/2012, at 6:56 PM, Patrik Modesto wrote:

Hi,

I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace
with RF=3 and one CF with two secondary indexes. I'm importing data
into this CF using Hadoop Mapreduce job, each row has less than 10
colkumns. From JMX:
MaxRowSize:  1597
MeanRowSize: 369

And there are some tens of millions of rows.

It's write-heavy usage and there is a big pressure on each node, there
are quite some dropped mutations on each node. After ~12 hours of
inserting I see these assertion exceptiona on 3 out of four nodes:

ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of
629444349 but now it is 588008950
       at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388)
       at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256)
       at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84)
       at
org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437)
       at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of
629444349 but now it is 588008950
       at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
       at java.util.concurrent.FutureTask.get(FutureTask.java:83)
       at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384)
       ... 7 more
Caused by: java.lang.AssertionError: originally calculated column size
of 629444349 but now it is 588008950
       at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124)
       at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
       at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161)
       at
org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380)
       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
       ... 3 more


Few lines regarding Hints from the output.log:

INFO 06:21:26,202 Compacting large row
system/HintsColumnFamily:70000000000000000000000000000000 (1712834057
bytes) incrementally
INFO 06:22:52,610 Compacting large row
system/HintsColumnFamily:10000000000000000000000000000000 (2616073981
bytes) incrementally
INFO 06:22:59,111 flushing high-traffic column family
CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated
305147360 bytes)
INFO 06:22:59,813 Enqueuing flush of
Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live
bytes, 7452 ops)
INFO 06:22:59,814 Writing
Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live
bytes, 7452 ops)