cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: Errors During Compaction
Date Wed, 26 Jan 2011 00:55:46 GMT
Limited joy I would say :)  No long term damage at least.

 

I ended up deleting (moving to another disk) all the sstables which fixed the problem. I ran
in to even more problems during repair (detailed in another recent email) but it seems to
have worked regardless. Just to be safe, I am in the process of starting a ‘manual repair’
(copying SSTables from other nodes for this particular CF then restarting and running a cleanup
+ major compaction).

 

Any thoughts on what the root cause of this problem could be? It is somewhat worrying that
a CF can randomly become corrupt bringing down the whole node. Cassandras handling of a corrupt
CF (regardless of how rare an occurrence) is less than elegant. 

 

Dan

 

From: Aaron Morton [mailto:aaron@thelastpickle.com] 
Sent: January-25-11 16:03
To: user@cassandra.apache.org
Subject: Re: Errors During Compaction

 

Dan how did you go with this? More joy, less joy or a continuation of the current level of
joy?

 

Aaron

 


On 24/01/2011, at 9:38 AM, Dan Hendry <dan.hendry.junk@gmail.com> wrote:

I have run into a strange problem and was hoping for suggestions on how to fix it (0.7.0).
When compaction occurs on one node for what appears to be one specific column family, the
following error pops up the Cassandra log. Compaction apparently fails and temp files don’t
get cleaned up. After a while and what seems to be multiple failed compactions on the CF,
the node runs out of disk space and crashes. Not sure if it is a related problem or a function
of this being a heavily used column family but after failing to compact, compaction restarts
on the same CF exacerbating the issue.

 

Problems with this specific node started earlier this weekend when it crashed with and OOM
error. This is quite surprising since my memtable thresholds and GC settings have been tuned
to run with quite a bit of overhead during normal operation (max heap usage usually <=
10 GB on a 12 GB heap, average usage of 6-8 GB). I could not find anything abnormal in the
logs which would prompt an OOM.

 

I will look things over tomorrow and try to provide a bit more information on the problem
but as a solution, I was going to wipe out all SSTables for this CF on this node and then
run a repair. Far from ideal, is this a reasonable solution?

 

 

ERROR [CompactionExecutor:1] 2011-01-23 14:10:29,855 AbstractCassandraDaemon.java (line 91)
Fatal exception in thread Thread[CompactionExecutor:1,1,RMI Runtime]

java.io.IOError: java.io.EOFException: attempted to skip -1983579368 bytes but only skipped
0

        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78)

        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)

        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:143)

        at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:135)

        at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)

        at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)

        at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)

        at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)

        at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)

        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)

        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)

        at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)

        at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)

        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)

        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)

        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.io.EOFException: attempted to skip -1983579368 bytes but only skipped 0

        at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:52)

        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)

        ... 20 more

 

Dan Hendry

(403) 660-2297

 

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.872 / Virus Database: 271.1.1/3402 - Release Date: 01/25/11 02:34:00


Mime
View raw message