Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 328767524 for ; Tue, 29 Nov 2011 07:00:50 +0000 (UTC) Received: (qmail 66054 invoked by uid 500); 29 Nov 2011 07:00:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 66021 invoked by uid 500); 29 Nov 2011 07:00:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 66011 invoked by uid 99); 29 Nov 2011 07:00:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Nov 2011 07:00:46 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of watcherfr@gmail.com designates 209.85.215.172 as permitted sender) Received: from [209.85.215.172] (HELO mail-ey0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Nov 2011 07:00:40 +0000 Received: by eabm6 with SMTP id m6so3055571eab.31 for ; Mon, 28 Nov 2011 23:00:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=+hM/fPDYT9twPFmarMAJ6t5inzVzc+kPEmzPw/XRgSo=; b=nKseF9gbDP0zP+75uQcfxQzj+gso+vdnb8TlOu+2glweJTMN+yI+z9gPlyhvZTRZao oqHvS9HrIX3WuYo/ICG0Aa/01V0mbOsWPVfo8PnIZvMI9iWupIu2tRio7lysUmDAZ1sa ffvRB1AWbK1H4Yg7XaPEyVjBuKJEel7gAE970= MIME-Version: 1.0 Received: by 10.180.0.100 with SMTP id 4mr46667164wid.48.1322550019000; Mon, 28 Nov 2011 23:00:19 -0800 (PST) Received: by 10.180.83.70 with HTTP; Mon, 28 Nov 2011 23:00:18 -0800 (PST) Date: Tue, 29 Nov 2011 08:00:18 +0100 Message-ID: Subject: Corrupt (negative) value length encountered From: Philippe To: user Content-Type: multipart/alternative; boundary=bcaec52c623314889704b2da2c2d --bcaec52c623314889704b2da2c2d Content-Type: text/plain; charset=ISO-8859-1 Hello, While running a cleanup, Cassandra stopped with the following exception and inspecting the logs revealed several exceptions such as below over the past 3 days.Given that it's dying on compactions, I'm really worried. If a row was trashed, will the error propagate from node to node or will it be localized and a repair will fix it ? Thanks java.io.IOError: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0 at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265) at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281) at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236) at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265) at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281) at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236) at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 113362603 but 157851071 requested at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265) at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281) at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236) at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) at java.util.concurrent.ConcurrentSkipListMap.(ConcurrentSkipListMap.java:1443) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:445) java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) != DecoratedKey(Token(bytes[02c67a8a54000000005311]), 02c67a8a54000000005311) in /var/lib/cassandra/data/XXXXXX-g-159-Data.db at org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:59) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1407) at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1274) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1314) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1261) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58) at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:105) --bcaec52c623314889704b2da2c2d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello,
While running a cleanup, Cassandra stopped with the following ex= ception and inspecting the logs revealed several exceptions such as below o= ver the past 3 days.Given that it's dying on compactions, I'm reall= y worried.

If a row was trashed, will the error propagate from nod= e to node or will it be localized and a repair will fix it ?=A0
<= br>
Thanks

java.io.IOError: or= g.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid colu= mn name length 0
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.ColumnIterator.deseria= lizeNext(ColumnSortedMap.java:265)
=A0 =A0 =A0 =A0 at org.apache.= cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
= =A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.ColumnIterator.next(ColumnS= ortedMap.java:236)
=A0 =A0 =A0 =A0 at java.util.concurrent.ConcurrentSkipListMap.buildFro= mSorted(ConcurrentSkipListMap.java:1493)

jav= a.io.IOError: java.io.IOException: Corrupt (negative) value length encounte= red
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.ColumnIterator.de= serializeNext(ColumnSortedMap.java:265)
=A0 =A0 =A0 =A0 at org.ap= ache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
<= div>=A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.ColumnIterator.next(Col= umnSortedMap.java:236)
=A0 =A0 =A0 =A0 at java.util.concurrent.ConcurrentSkipListMap.buildFro= mSorted(ConcurrentSkipListMap.java:1493)

java.io.IOError: java.io.IOException: mmap segment underflow; remaining i= s 113362603 but 157851071 requested
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.ColumnIterator.deseria= lizeNext(ColumnSortedMap.java:265)
=A0 =A0 =A0 =A0 at org.apache.= cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
= =A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.ColumnIterator.next(ColumnS= ortedMap.java:236)
=A0 =A0 =A0 =A0 at java.util.concurrent.ConcurrentSkipListMap.buildFro= mSorted(ConcurrentSkipListMap.java:1493)
=A0 =A0 =A0 =A0 at java.= util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.ja= va:1443)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.SuperColumnSerializer.deser= ialize(SuperColumn.java:445)
java.lang.AssertionError:= DecoratedKey(Token(bytes[]), ) !=3D DecoratedKey(Token(bytes[02c67a8a54000= 000005311]), 02c67a8a54000000005311) in /var/lib/cassandra/data/XXXXXX-g-15= 9-Data.db
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.columniterator.SSTableSlice= Iterator.<init>(SSTableSliceIterator.java:59)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIter= ator(SliceQueryFilter.java:66)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.filter.QueryFilter.getSSTab= leColumnIterator(QueryFilter.java:80)
=A0 =A0 =A0 =A0 at org.apac= he.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java= :1407)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(= ColumnFamilyStore.java:1274)
=A0 =A0 =A0 =A0 at org.apache.cassan= dra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1314)
=
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnFamilyStore.getColumn= Family(ColumnFamilyStore.java:1261)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.Table.getRow(Table.java:385= )
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.SliceByNamesReadComm= and.getRow(SliceByNamesReadCommand.java:58)
=A0 =A0 =A0 =A0 at or= g.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutati= on.java:105)


--bcaec52c623314889704b2da2c2d--