incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David McNelis <dmcne...@gmail.com>
Subject FSReadError
Date Mon, 09 Sep 2013 13:15:08 GMT
Morning,

I'm getting the following error (21 node cluster running 1.2.8)

FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
  at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:93)
	at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:312)
	at org.apache.cassandra.io.sstable.SSTableScanner.seekTo(SSTableScanner.java:98)
	at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:73)
	at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1424)
	at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1476)
	at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:46)
	at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:58)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Input/output error
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
	at sun.nio.ch.IOUtil.read(IOUtil.java:198)
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)
	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:108)
	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85)
	... 11 more

This starts during compaction and compaction ends up stuck, the node
eventually winds up locked up with this error repeating throughout the
logs, and requires a restart.

On restart I tried running a nodetool scrub against the offending
column family, which itself failed.

Any suggestions on taking care of this? Should I just delete the
sstable and repair? The node keeps starting compactions and running
into this after a period of time.


David

Mime
View raw message