incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ondřej Černoš <cern...@gmail.com>
Subject Re: Can't perform repair on a 1.1.5 cassandra node -SSTable corrupted
Date Wed, 07 Aug 2013 15:07:46 GMT
Hello,

please see these issues:
https://issues.apache.org/jira/browse/CASSANDRA-5686 and
https://issues.apache.org/jira/browse/CASSANDRA-5391 if you hit any of them.

regards,

ondrej cernos


On Wed, Aug 7, 2013 at 5:00 PM, Madalina Matei <madalinaimatei@gmail.com>wrote:

> Hi,
>
> I have a 5 nodes cassandra (version 1.1.5) ring, RF=2, CL- READ/Write =1.
> After a node went down without any error reported in OS syslog or Cassandra
> syslog i decided to perform a repair.
>
> Each time i run a nodetool repair I get this error:
>
>  INFO [FlushWriter:5] 2013-08-07 11:09:26,770 Memtable.java (line 305)
> Completed flushing /data/<data>-298-Data.db (18694 bytes) for commitlog
> position ReplayPosition(segmentId=1375867548785, position=199)
> ERROR [Thrift:286] 2013-08-07 11:10:04,448 CustomTThreadPoolServer.java
> (line 204) Error occurred during processing of message.
> java.lang.RuntimeException: error reading 1 of 1
>     at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
>     at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
>     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>     at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
>     at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:203)
>     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>     at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117)
>     at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140)
>     at
> org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:107)
>     at
> org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:80)
>     at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
>     at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
>     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>     at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
>     at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
>     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
>     at
> org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
>     at
> org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
>     at
> org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
>     at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
>     at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
>     at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
>     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>     at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
>     at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
>     at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
>     at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
>     at
> org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
>     at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
>     at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
>     at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
>     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
>     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>     at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
>     at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
>     at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
>     at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
>     at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
>     at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
>     at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
>     at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
>     ... 32 more
>
>
> I performed first a sstablescrub and i got the same error.
>
> The problem is that my map-reduce job fails each time throwing in this
> specific node syslog an error complaining about the same SSTable.
>
> ERROR [Thrift:50] 2013-08-07 09:30:40,103 CustomTThreadPoolServer.java
> (line 204) Error occurred during processing of message.
> java.lang.RuntimeException: SSTableScanner(file=/data/<data>-298-Data.db-
> chunk length 65536, data length 773855911.
> sstable=SSTableReader(path='/data/<data>-298-Data.db') exhausted=false)
> failed to provide next columns from
> KeyScanningIterator(finishedAt:105054114)
>     at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:198)
>     at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:151)
>     at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:143)
>     at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
>     at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:149)
>     at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126)
>     at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
>     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>     at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
>     at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
>     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
>     at
> org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
>     at
> org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
>     at
> org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
>     at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
>     at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
>     at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
>     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>     at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
>     at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
>     at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
>     at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
>     at
> org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
>     at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
>     at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
>     at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
>     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
>     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>     at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
>     at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
>     at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
>     at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)
>
>
> Is there any way to fix that specific SSTable or any other recommendations?
>
> Thanks,
> Madalina
>
>

Mime
View raw message