Hi,
We are running a 8 node cassandra cluster. We do reads and writes
with CL QUORUM. During our performance test last weekend we noticed
compaction pending stats started growing after 10 hrs of testing.
System log showed errors during compaction. I am attaching the
system.log file. Is this a known issue?
thanks
Ramesh
java.io.IOError: java.io.IOException: dataSize of 15625949818928
starting at 6767047 would be larger than file
/var/lib/cassandra/data/MSA/modseq-h-12758-Data.db length 10486216
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:153)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:85)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:142)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:134)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:37)
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:132)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
|