Some more data on this.  It appears to happen specifically on large row compactions.  My column families incorporate the new maps, and the maps get updated over time.  could this be a bug?  I previously ran into this:

 https://issues.apache.org/jira/browse/CASSANDRA-5677

and am running bits with this patch ( basically I'm running head on the 1.2 branch).

Any info would be helpful.

Thanks!

Paul

a more recent log, with the compaction start, as you can see I seem to be under a bit of heap pressure as well:

 INFO [CompactionExecutor:4] 2013-07-19 19:19:16,776 CompactionController.java (line 156) Compacting large row fanzo/tweets_by_team:philadelphia-eagles (134039720 bytes) incrementally
 WARN [ScheduledTasks:1] 2013-07-19 19:19:37,824 GCInspector.java (line 142) Heap is 0.797306136781001 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2013-07-19 19:19:37,826 StorageService.java (line 3618) Flushing CFS(Keyspace='fanzo', ColumnFamily='tweets_by_team') to relieve memory pressure
 INFO [ScheduledTasks:1] 2013-07-19 19:19:37,826 ColumnFamilyStore.java (line 630) Enqueuing flush of Memtable-tweets_by_team@296696472(11236776/11236776 serialized/live bytes, 5596 ops)
 INFO [FlushWriter:9] 2013-07-19 19:19:37,827 Memtable.java (line 461) Writing Memtable-tweets_by_team@296696472(11236776/11236776 serialized/live bytes, 5596 ops)
 INFO [FlushWriter:9] 2013-07-19 19:19:39,143 Memtable.java (line 495) Completed flushing /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-ic-6880-Data.db (2237487 bytes) for commitlog position ReplayPosition(segmentId=1374260151415, position=10223602)
ERROR [CompactionExecutor:4] 2013-07-19 19:19:51,969 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.AssertionError: incorrect row data size 26781347 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-6879-Data.db; correct is 27028794
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)



On Jul 18, 2013, at 11:53 AM, Paul Ingalls <paulingalls@gmail.com> wrote:

I'm seeing quite a few of these on pretty much all of the nodes of my 1.2.6 cluster.  Is this something I should be worried about?  If so, do I need to run upgradesstables or run a scrub?

ERROR [CompactionExecutor:4] 2013-07-18 18:49:02,609 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.AssertionError: incorrect row data size 72128792 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_affiliation/fanzo-tweets_by_affiliation-tmp-ic-918-Data.db; correct is 72148465 at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)

Thanks!

Paul