incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject IOException running cassandra 1.0.5
Date Fri, 09 Dec 2011 12:37:33 GMT
Hi,

 I have a 30 node cassandra cluster running on RHEL6 64 bit. RF=3,
reads and  writes performed with QUORUM. After few hours of test run,
I am seeing this error in the system.log file.

[root@MSA-VM-18 cassandra]# cat
/var/lib/cassandra/data/MSA/modseq-hb-419-Digest.sha1
71e43a932a29553720149bb4f93727e4d269735d
modseq-hb-419-Data.db[root@MSA-VM-18 cassandra]#
[root@MSA-VM-18 cassandra]# sha1sum
/var/lib/cassandra/data/MSA/modseq-hb-419-Data.db
033f5aea5590851377d3bb79df27f0e6eedb6b95
/var/lib/cassandra/data/MSA/modseq-hb-419-Data.db
[root@MSA-VM-18 cassandra]#

Any pointers to troubleshoot this issue?

I am attaching the system.log file for your reference.

thanks
Ramesh


 INFO [CompactionExecutor:296] 2011-12-09 04:36:40,430
CompactionTask.java (line 112) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/transactions-hb-55-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/transactions-hb-53-Data.db')]
 INFO [CompactionExecutor:296] 2011-12-09 04:36:40,501
CompactionTask.java (line 213) Compacted to
[/var/lib/cassandra/data/MSA/transactions-hb-56-Data.db,].  280,210 to
144,785 (~51% of original) bytes for 3 keys at 2.191710MB/s.  Time:
63ms.
ERROR [CompactionExecutor:295] 2011-12-09 04:36:41,320
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:295,1,main]
java.io.IOError: java.io.IOException: dataSize of 14293651161088
starting at 5541742 would be larger than file
/var/lib/cassandra/data/MSA/modseq-hb-419-Data.db length 10486511
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:154)
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:86)
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:70)
        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177)
        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:142)
        at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:134)
        at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:37)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
        at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: dataSize of 14293651161088 starting at
5541742 would be larger than file
/var/lib/cassandra/data/MSA/modseq-hb-419-Data.db length 10486511
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:115)
        ... 23 more
 INFO [COMMIT-LOG-WRITER] 2011-12-09 04:36:52,668
CommitLogSegment.java (line 60) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1323405412668.log

Mime
View raw message