incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madalina Matei <madalinaima...@gmail.com>
Subject Can't perform repair on a 1.1.5 cassandra node -SSTable corrupted
Date Wed, 07 Aug 2013 15:00:48 GMT
Hi,

I have a 5 nodes cassandra (version 1.1.5) ring, RF=2, CL- READ/Write =1.
After a node went down without any error reported in OS syslog or Cassandra
syslog i decided to perform a repair.

Each time i run a nodetool repair I get this error:

 INFO [FlushWriter:5] 2013-08-07 11:09:26,770 Memtable.java (line 305)
Completed flushing /data/<data>-298-Data.db (18694 bytes) for commitlog
position ReplayPosition(segmentId=1375867548785, position=199)
ERROR [Thrift:286] 2013-08-07 11:10:04,448 CustomTThreadPoolServer.java
(line 204) Error occurred during processing of message.
java.lang.RuntimeException: error reading 1 of 1
    at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
    at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
    at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
    at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:203)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
    at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117)
    at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140)
    at
org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:107)
    at
org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:80)
    at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
    at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
    at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
    at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
    at
org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
    at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
    at
org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
    at
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
    at
org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
    at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
    at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
    at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
    at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
    at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
    at
org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
    at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
    at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
    at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
    at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
    at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
    at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
    at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
    at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
    ... 32 more


I performed first a sstablescrub and i got the same error.

The problem is that my map-reduce job fails each time throwing in this
specific node syslog an error complaining about the same SSTable.

ERROR [Thrift:50] 2013-08-07 09:30:40,103 CustomTThreadPoolServer.java
(line 204) Error occurred during processing of message.
java.lang.RuntimeException: SSTableScanner(file=/data/<data>-298-Data.db-
chunk length 65536, data length 773855911.
sstable=SSTableReader(path='/data/<data>-298-Data.db') exhausted=false)
failed to provide next columns from
KeyScanningIterator(finishedAt:105054114)
    at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:198)
    at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:151)
    at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:143)
    at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
    at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:149)
    at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126)
    at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
    at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
    at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
    at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
    at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
    at
org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
    at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
    at
org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
    at
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
    at
org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
    at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
    at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
    at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
    at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
    at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
    at
org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
    at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
    at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
    at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
    at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
    at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
    at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
    at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)


Is there any way to fix that specific SSTable or any other recommendations?

Thanks,
Madalina

Mime
View raw message