Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BEBF86329 for ; Sat, 18 Jun 2011 14:54:09 +0000 (UTC) Received: (qmail 94410 invoked by uid 500); 18 Jun 2011 14:54:09 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 94372 invoked by uid 500); 18 Jun 2011 14:54:09 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 94237 invoked by uid 99); 18 Jun 2011 14:54:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 14:54:09 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 14:54:07 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id D18AF420400 for ; Sat, 18 Jun 2011 14:53:47 +0000 (UTC) Date: Sat, 18 Jun 2011 14:53:47 +0000 (UTC) From: "Jonathan Ellis (JIRA)" To: commits@cassandra.apache.org Message-ID: <653973504.18133.1308408827855.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1736752540.18122.1308407447541.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Resolved] (CASSANDRA-2793) SSTable "Corrupt (negative) value length encountered" exception blocks compaction. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2793. --------------------------------------- Resolution: Not A Problem Fix Version/s: (was: 0.7.7) Did you miss this line? bq. Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6994-Data.db') complete > SSTable "Corrupt (negative) value length encountered" exception blocks compaction. > ---------------------------------------------------------------------------------- > > Key: CASSANDRA-2793 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2793 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.7.6 > Environment: Ubuntu > Reporter: Dominic Williams > > A node was consistently experiencing high CPU load. Examination of the logs showed that compaction of an sstable was failing with an error: > INFO [CompactionExecutor:1] 2011-06-17 00:18:51,676 CompactionManager.java (line 395) Compacting [SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6993-Data.db'),SSTableReader( > path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6994-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6995-Data.db'),SSTableReader(path='/var/opt/cassandra > /data/FightMyMonster/UserMonsters-f-6996-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6998-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/Use > rMonsters-f-7000-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7002-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7004-Data.db > '),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7006-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7008-Data.db'),SSTableReader(path='/ > var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7012-Data.db'),SSTableReader(path='/var/opt/cassandra/data/F > ightMyMonster/UserMonsters-f-7014-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7016-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonste > rs-f-7018-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7020-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7022-Data.db'),SSTa > bleReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7024-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7026-Data.db'),SSTableReader(path='/var/opt > /cassandra/data/FightMyMonster/UserMonsters-f-7028-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7030-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyM > onster/UserMonsters-f-7032-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7034-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-70 > 36-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7038-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7040-Data.db'),SSTableRead > er(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7042-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7044-Data.db'),SSTableReader(path='/var/opt/cassan > dra/data/FightMyMonster/UserMonsters-f-7046-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7048-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7050-Data.db'),SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7052-Data.db')] > ERROR [CompactionExecutor:1] 2011-06-17 00:19:21,446 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] > java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:252) > at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:268) > at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:227) at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) > at java.util.concurrent.ConcurrentSkipListMap.(ConcurrentSkipListMap.java:1443) > at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:379) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:362) > at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:322) > at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:201) > at org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:78) > at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:154) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:110) > at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:45) > at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:74) > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) > at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) > at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448) > at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124) > at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Corrupt (negative) value length encountered > at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:315) > at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:99) > at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:248) > ... 26 more > Scrub was run on the keyspace (as a last ditch measure) but this did not work: > INFO [CompactionExecutor:1] 2011-06-17 00:43:42,023 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7494-Data.db') > INFO [CompactionExecutor:1] 2011-06-17 00:43:43,317 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7494-Data.db') complete: 379 row > s in new sstable and 0 empty (tombstoned) rows dropped > INFO [CompactionExecutor:1] 2011-06-17 00:43:43,317 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6994-Data.db') > WARN [CompactionExecutor:1] 2011-06-17 00:43:44,516 CompactionManager.java (line 606) Non-fatal error reading row (stacktrace follows) > java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered > at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:252) > at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:268) > at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:227) > at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) > at java.util.concurrent.ConcurrentSkipListMap.(ConcurrentSkipListMap.java:1443) > at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:379) > at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:362) > at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:322) > at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129) > at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:201) > at org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:78) > at org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:783) > at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590) > at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) > at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Corrupt (negative) value length encountered > at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:315) > at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:99) > at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:248) > ... 19 more > WARN [CompactionExecutor:1] 2011-06-17 00:43:44,517 CompactionManager.java (line 640) Row at 9517800 is unreadable; skipping to next > INFO [CompactionExecutor:1] 2011-06-17 00:43:45,073 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6994-Data.db') complete: 1029 rows in new sstable and 0 empty (tombstoned) rows dropped > WARN [CompactionExecutor:1] 2011-06-17 00:43:45,073 CompactionManager.java (line 654) Unable to recover 1 rows that were skipped. You can attempt manual recovery from the pre-scrub snapshot. You can also run nodetool repair to transfer the data from a healthy replica, if any -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira