Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CC01E3CB1 for ; Thu, 28 Apr 2011 16:58:59 +0000 (UTC) Received: (qmail 45882 invoked by uid 500); 28 Apr 2011 16:58:57 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 45851 invoked by uid 500); 28 Apr 2011 16:58:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 45843 invoked by uid 99); 28 Apr 2011 16:58:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 16:58:57 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_FROM,HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of daniel.doubleday@gmx.net designates 213.165.64.23 as permitted sender) Received: from [213.165.64.23] (HELO mailout-de.gmx.net) (213.165.64.23) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 28 Apr 2011 16:58:50 +0000 Received: (qmail invoked by alias); 28 Apr 2011 16:58:27 -0000 Received: from p578bde86.dip0.t-ipconnect.de (EHLO caladan.smeet.de) [87.139.222.134] by mail.gmx.net (mp057) with SMTP; 28 Apr 2011 18:58:27 +0200 X-Authenticated: #3445653 X-Provags-ID: V01U2FsdGVkX1+6HYIzbma67ULW2el+d+2awJILpxZH52l8wg6V3J 08iTXQUIIyaVxo From: Daniel Doubleday Content-Type: multipart/alternative; boundary=Apple-Mail-1--714807980 Subject: Strange corrupt sstable Date: Thu, 28 Apr 2011 18:58:26 +0200 Message-Id: <10F3D309-279A-420E-9976-33F32C713FA5@gmx.net> To: user@cassandra.apache.org Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) X-Y-GMX-Trusted: 0 --Apple-Mail-1--714807980 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi all on one of our dev machines we ran into this: INFO [CompactionExecutor:1] 2011-04-28 15:07:35,174 SSTableWriter.java = (line 108) Last written key : = DecoratedKey(12707736894140473154801792860916528374, 74657374) INFO [CompactionExecutor:1] 2011-04-28 15:07:35,174 SSTableWriter.java = (line 109) Current key : = DecoratedKey(9639328335678452015784116654185025244, = 61c080c080c080c080c080c0803a60) INFO [CompactionExecutor:1] 2011-04-28 15:07:35,175 SSTableWriter.java = (line 110) Writing into file = /var/lib/cassandra/data/Smeet/Tracker-tmp-f-752-Data.db ERROR [CompactionExecutor:1] 2011-04-28 15:07:35,179 = AbstractCassandraDaemon.java (line 113) Fatal exception in thread = Thread[CompactionExecutor:1,1,main] java.io.IOException: Keys must be written in ascending order. at = org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.j= ava:111) at = org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:12= 8) at = org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.j= ava:451) at = org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:12= 4) at = org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94= ) at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1110) at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :603) at java.lang.Thread.run(Thread.java:636) After looking into this we found that one SSTable was corrupt: INFO 18:10:23,767 checking sstable = /Users/dd/Temp/cassandra/data/Smeet/Tracker-f-21-Data.db INFO 18:10:23,773 DecoratedKey(12707736894140473154801792860916528374, = 74657374) INFO 18:10:23,777 DecoratedKey(9639328335678452015784116654185025244, = 61c080c080c080c080c080c0803a60) INFO 18:10:23,781 DecoratedKey(14745077953608664589374317792467471221, = 7265706f2d333631343039) INFO 18:10:23,783 DecoratedKey(20959981347795136420050187639971803838, = 6576656e746c6f672d333631383838) INFO 18:10:23,785 DecoratedKey(21549594967496343960898734735584772470, = 6576656e746c6f672d333631383837) INFO 18:10:23,787 DecoratedKey(21604488955263763122920767208831708343, = 6576656e746c6f672d333631383835) INFO 18:10:23,790 DecoratedKey(50100329769829417461462475380721306089, = 61c080c080c080c080c080c0803a5f) INFO 18:10:23,792 DecoratedKey(46089976107344533463199626515706634206, = 7265706f2d333631303436) INFO 18:10:23,794 DecoratedKey(50825113246224326722027014732583502298, = 6c6f674576656e742d333631383431) INFO 18:10:23,797 DecoratedKey(146509745693074257726394585872920772980, = 61c080c080c080c080c080c0803a59) INFO 18:10:23,801 DecoratedKey(63662277073629697060427801149510996510, = 6c6f674576656e742d333631383432) INFO 18:10:23,803 DecoratedKey(79706531464725069255477768601476202854, = 6576656e746c6f672d333631383834) INFO 18:10:23,808 DecoratedKey(99662740527281958972877086277408797556, = 6c6f674576656e742d333631383430) INFO 18:10:23,810 DecoratedKey(24216004522287050570821311008920284627, = 61c080c080c080c080c080c0803a5a) INFO 18:10:23,816 DecoratedKey(130819691046705791898531759480356609485, = 61c080c080c080c080c080c0803a5e) INFO 18:10:24,121 DecoratedKey(107312182597869061361893705497084751672, = 61c080c080c080c080c080c0803a58) INFO 18:10:24,123 DecoratedKey(142873550623537979564886292314205248041, = 7265706f2d333631363731) INFO 18:10:24,124 DecoratedKey(147476468396567670371575440897348610391, = 6576656e746c6f672d333631383836) I couldn't figure out how SSTableWriter could possible write such a file = so ... Just wanted to report. Maybe it rings a bell somewhere. It seems really strange that some data corruption would leave the file = readable for the SSTableReader. History of this server is 0.6.5 -> 0.7.4. But this file was obviously = written by 0.7.4 Cheers, Daniel --Apple-Mail-1--714807980 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi = all

on one of our dev machines we ran into = this:

INFO [CompactionExecutor:1] 2011-04-28 15:07:35,174 = SSTableWriter.java (line 108) Last written key : = DecoratedKey(12707736894140473154801792860916528374, 74657374)
 INFO [CompactionExecutor:1] = 2011-04-28 15:07:35,175 SSTableWriter.java (line 110) Writing into file = /var/lib/cassandra/data/Smeet/Tracker-tmp-f-752-Data.db
<= span class=3D"Apple-style-span" style=3D"font-family: 'Lucida Grande'; = color: rgb(20, 20, 20); line-height: 14px; ">ERROR = [CompactionExecutor:1] 2011-04-28 15:07:35,179 = AbstractCassandraDaemon.java (line 113) Fatal exception in thread = Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be = written in ascending order.
        at = org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.j= ava:111)
        at = org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:12= 8)
        at = org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.j= ava:451)
        at = org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:12= 4)
        at = org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94= )
        at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :603)
        at = java.lang.Thread.run(Thread.java:636)


After looking into this we found that one SSTable was = corrupt:

INFO = 18:10:23,767 checking sstable = /Users/dd/Temp/cassandra/data/Smeet/Tracker-f-21-Data.db
 I= NFO 18:10:23,773 DecoratedKey(12707736894140473154801792860916528374, = 74657374)
 INFO 18:10:23,777 = DecoratedKey(9639328335678452015784116654185025244, = 61c080c080c080c080c080c0803a60)
 INFO 18:10:23,781 = DecoratedKey(14745077953608664589374317792467471221, = 7265706f2d333631343039)
 INFO 18:10:23,783 = DecoratedKey(20959981347795136420050187639971803838, = 6576656e746c6f672d333631383838)
 INFO 18:10:23,785 = DecoratedKey(21549594967496343960898734735584772470, = 6576656e746c6f672d333631383837)
 INFO 18:10:23,787 = DecoratedKey(21604488955263763122920767208831708343, = 6576656e746c6f672d333631383835)
 INFO 18:10:23,790 = DecoratedKey(50100329769829417461462475380721306089, = 61c080c080c080c080c080c0803a5f)
 INFO 18:10:23,792 = DecoratedKey(46089976107344533463199626515706634206, = 7265706f2d333631303436)
 INFO 18:10:23,794 = DecoratedKey(50825113246224326722027014732583502298, = 6c6f674576656e742d333631383431)
 INFO 18:10:23,797 = DecoratedKey(146509745693074257726394585872920772980, = 61c080c080c080c080c080c0803a59)
 INFO 18:10:23,801 = DecoratedKey(63662277073629697060427801149510996510, = 6c6f674576656e742d333631383432)
 INFO 18:10:23,803 = DecoratedKey(79706531464725069255477768601476202854, = 6576656e746c6f672d333631383834)
 INFO 18:10:23,808 = DecoratedKey(99662740527281958972877086277408797556, = 6c6f674576656e742d333631383430)
 INFO 18:10:23,810 = DecoratedKey(24216004522287050570821311008920284627, = 61c080c080c080c080c080c0803a5a)
 INFO 18:10:23,816 = DecoratedKey(130819691046705791898531759480356609485, = 61c080c080c080c080c080c0803a5e)
 INFO 18:10:24,121 = DecoratedKey(107312182597869061361893705497084751672, = 61c080c080c080c080c080c0803a58)
 INFO 18:10:24,123 = DecoratedKey(142873550623537979564886292314205248041, = 7265706f2d333631363731)
 INFO 18:10:24,124 = DecoratedKey(147476468396567670371575440897348610391, = 6576656e746c6f672d333631383836)

I couldn't = figure out how SSTableWriter could possible write such a file so ... = Just wanted to report. Maybe it rings a bell = somewhere.

It seems really strange that some = data corruption would leave the file readable for the = SSTableReader.

History of this server is 0.6.5 = -> 0.7.4. But this file was obviously written by = 0.7.4



Cheers,
=
Daniel


= = --Apple-Mail-1--714807980--