Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6A226200B29 for ; Thu, 30 Jun 2016 16:49:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 68F0E160A63; Thu, 30 Jun 2016 14:49:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 73257160A52 for ; Thu, 30 Jun 2016 16:49:11 +0200 (CEST) Received: (qmail 33257 invoked by uid 500); 30 Jun 2016 14:49:10 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 33211 invoked by uid 99); 30 Jun 2016 14:49:10 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2016 14:49:10 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 3AA2A2C029F for ; Thu, 30 Jun 2016 14:49:10 +0000 (UTC) Date: Thu, 30 Jun 2016 14:49:10 +0000 (UTC) From: "Paulo Motta (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-12113) Cassandra 3.5.0 Repair Error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 30 Jun 2016 14:49:12 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-12113?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-12113: ------------------------------------ Assignee: Paulo Motta Priority: Major (was: Critical) Thanks for the report! While the exception message can be a bit scary, ther= e isn't data corruption going on so I will reduce the priority of this to m= ajor. After CASSANDRA-10797, we finish the {{SSTableWriters}} as soon as they are= received, but we don't reference the {{SSTableReaders}} until the stream s= ession is finished, so their index summaries can be redistributed before th= e stream session is finished causing the {{Memory was freed}} error, and su= bsequent {{CorruptSSTableException}}. The fix here is to reference the {{SSTableReaders}} as soon as they are cre= ated to prevent old references from being released earlier. > Cassandra 3.5.0 Repair Error > ---------------------------- > > Key: CASSANDRA-12113 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1211= 3 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Production > Reporter: Serhat R=C4=B1fat Demircan > Assignee: Paulo Motta > > I got the following error while repairing nodes with the "nodetool repair= " command. Error occured on 2 nodes in the cluster which have 9 nodes. > =20 > Interesting thing is corrupted sstable is no more exists one of 2 nodes. = Copied existing one to test cluster and restored table from that sstable. N= o error occured on test cluster. > {noformat} > ERROR [StreamReceiveTask:6] 2016-06-16 02:56:47,480 StreamReceiveTask.jav= a:215 - Error applying streamed data: > java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.= apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/ca= ssandra/data/keyspace/table/ma-1518-big-Data.db > at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.jav= a:50) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtiliti= es.java:372) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.index.SecondaryIndexManager.buildIndexesB= locking(SecondaryIndexManager.java:375) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.index.SecondaryIndexManager.buildAllIndex= esBlocking(SecondaryIndexManager.java:262) ~[apache-cassandra-3.5.0.jar:3.5= .0] > at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionR= unnable.run(StreamReceiveTask.java:182) ~[apache-cassandra-3.5.0.jar:3.5.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:511) [na:1.8.0_91] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1= .8.0_91] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1142) [na:1.8.0_91] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:617) [na:1.8.0_91] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.= io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keys= pace/table/ma-1518-big-Data.db > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [n= a:1.8.0_91] > at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1= .8.0_91] > at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtiliti= es.java:365) ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 8 common frames omitted > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corru= pted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db > at org.apache.cassandra.db.columniterator.AbstractSSTableIterator= $Reader.hasNext(AbstractSSTableIterator.java:367) ~[apache-cassandra-3.5.0.= jar:3.5.0] > at org.apache.cassandra.db.columniterator.AbstractSSTableIterator= .hasNext(AbstractSSTableIterator.java:229) ~[apache-cassandra-3.5.0.jar:3.5= .0] > at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext= (SSTableIterator.java:32) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIte= rator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache= -cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBou= nd.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) ~[apache-cassan= dra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBou= nd.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) ~[apache-cassan= dra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIt= erator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.MergeIterator$Candidate.advance(Mer= geIterator.java:374) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(Mer= geIterator.java:186) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext= (MergeIterator.java:155) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIt= erator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.UnfilteredRowIterators$Unfiltered= RowMergeIterator.computeNext(UnfilteredRowIterators.java:419) ~[apache-cass= andra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.UnfilteredRowIterators$Unfiltered= RowMergeIterator.computeNext(UnfilteredRowIterators.java:279) ~[apache-cass= andra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIt= erator.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.ja= va:112) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.index.SecondaryIndexManager.indexPartitio= n(SecondaryIndexManager.java:549) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:= 570) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.b= uild(CollatedViewIndexBuilder.java:70) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.compaction.CompactionManager$12.run(Co= mpactionManager.java:1472) ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 5 common frames omitted > Caused by: java.io.IOException: Error building row with data deserialized= from /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db - chunk le= ngth 65536, data length 85398996. > at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeR= owBody(UnfilteredSerializer.java:446) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserial= izer.readNext(UnfilteredDeserializer.java:211) ~[apache-cassandra-3.5.0.jar= :3.5.0] > at org.apache.cassandra.db.columniterator.SSTableIterator$Forward= Reader.computeNext(SSTableIterator.java:125) ~[apache-cassandra-3.5.0.jar:3= .5.0] > at org.apache.cassandra.db.columniterator.SSTableIterator$Forward= Reader.hasNextInternal(SSTableIterator.java:149) ~[apache-cassandra-3.5.0.j= ar:3.5.0] > at org.apache.cassandra.db.columniterator.AbstractSSTableIterator= $Reader.hasNext(AbstractSSTableIterator.java:354) ~[apache-cassandra-3.5.0.= jar:3.5.0] > ... 23 common frames omitted > Caused by: java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory= .java:103) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~= [apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(= CompressionMetadata.java:235) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.io.compress.CompressedRandomAccessReader.= reBufferMmap(CompressedRandomAccessReader.java:170) ~[apache-cassandra-3.5.= 0.jar:3.5.0] > at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(Rando= mAccessReader.java:111) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.io.util.RebufferingInputStream.readByte(R= ebufferingInputStream.java:144) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VI= ntCoding.java:63) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.io.util.RebufferingInputStream.readUnsign= edVInt(RebufferingInputStream.java:209) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.SerializationHeader.readLocalDeletionT= ime(SerializationHeader.java:210) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.= java:224) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleCo= lumn(UnfilteredSerializer.java:455) ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeR= owBody(UnfilteredSerializer.java:433) ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 27 common frames omitted > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)