incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Ingalls <paulinga...@gmail.com>
Subject Re: Incorrect row data size
Date Fri, 19 Jul 2013 19:22:25 GMT
Some more data on this.  It appears to happen specifically on large row compactions.  My column
families incorporate the new maps, and the maps get updated over time.  could this be a bug?
 I previously ran into this:

 https://issues.apache.org/jira/browse/CASSANDRA-5677

and am running bits with this patch ( basically I'm running head on the 1.2 branch)….

Any info would be helpful.

Thanks!

Paul

a more recent log, with the compaction start, as you can see I seem to be under a bit of heap
pressure as well:

 INFO [CompactionExecutor:4] 2013-07-19 19:19:16,776 CompactionController.java (line 156)
Compacting large row fanzo/tweets_by_team:philadelphia-eagles (134039720 bytes) incrementally
 WARN [ScheduledTasks:1] 2013-07-19 19:19:37,824 GCInspector.java (line 142) Heap is 0.797306136781001
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2013-07-19 19:19:37,826 StorageService.java (line 3618) Flushing
CFS(Keyspace='fanzo', ColumnFamily='tweets_by_team') to relieve memory pressure
 INFO [ScheduledTasks:1] 2013-07-19 19:19:37,826 ColumnFamilyStore.java (line 630) Enqueuing
flush of Memtable-tweets_by_team@296696472(11236776/11236776 serialized/live bytes, 5596 ops)
 INFO [FlushWriter:9] 2013-07-19 19:19:37,827 Memtable.java (line 461) Writing Memtable-tweets_by_team@296696472(11236776/11236776
serialized/live bytes, 5596 ops)
 INFO [FlushWriter:9] 2013-07-19 19:19:39,143 Memtable.java (line 495) Completed flushing
/mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-ic-6880-Data.db
(2237487 bytes) for commitlog position ReplayPosition(segmentId=1374260151415, position=10223602)
ERROR [CompactionExecutor:4] 2013-07-19 19:19:51,969 CassandraDaemon.java (line 192) Exception
in thread Thread[CompactionExecutor:4,1,main]
java.lang.AssertionError: incorrect row data size 26781347 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-6879-Data.db;
correct is 27028794
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)



On Jul 18, 2013, at 11:53 AM, Paul Ingalls <paulingalls@gmail.com> wrote:

> I'm seeing quite a few of these on pretty much all of the nodes of my 1.2.6 cluster.
 Is this something I should be worried about?  If so, do I need to run upgradesstables or
run a scrub?
> 
> ERROR [CompactionExecutor:4] 2013-07-18 18:49:02,609 CassandraDaemon.java (line 192)
Exception in thread Thread[CompactionExecutor:4,1,main] 
> java.lang.AssertionError: incorrect row data size 72128792 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_affiliation/fanzo-tweets_by_affiliation-tmp-ic-918-Data.db;
correct is 72148465 at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)

> at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)

> at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)

> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)

> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)

> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)

> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

> at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at
java.lang.Thread.run(Thread.java:724)
> 
> Thanks!
> 
> Paul


Mime
View raw message