cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serhat Rıfat Demircan (JIRA) <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-12113) Cassandra 3.5.0 Repair Error
Date Thu, 30 Jun 2016 13:15:10 GMT
Serhat Rıfat Demircan created CASSANDRA-12113:
-------------------------------------------------

             Summary: Cassandra 3.5.0 Repair Error
                 Key: CASSANDRA-12113
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12113
             Project: Cassandra
          Issue Type: Bug
          Components: Compaction
         Environment: Production
            Reporter: Serhat Rıfat Demircan
            Priority: Critical


I got the following error while repairing nodes with the "nodetool repair" command. Error
occured on 2 nodes in the cluster which have 9 nodes.
 
Interesting thing is corrupted sstable is no more exists one of 2 nodes. Copied existing one
to test cluster and restored table from that sstable. No error occured on test cluster.

{noformat}
ERROR [StreamReceiveTask:6] 2016-06-16 02:56:47,480 StreamReceiveTask.java:215 - Error applying
streamed data:
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException:
Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
        at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:375)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:262)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:182)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException:
Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_91]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_91]
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365) ~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 8 common frames omitted
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 5 common frames omitted
Caused by: java.io.IOException: Error building row with data deserialized from /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
- chunk length 65536, data length 85398996.
        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 23 common frames omitted
Caused by: java.lang.AssertionError: Memory was freed
        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433)
~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 27 common frames omitted
{noformat}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message