Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7AAEB1048F for ; Mon, 5 Aug 2013 15:24:30 +0000 (UTC) Received: (qmail 17500 invoked by uid 500); 5 Aug 2013 15:24:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 17479 invoked by uid 500); 5 Aug 2013 15:24:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 17470 invoked by uid 99); 5 Aug 2013 15:24:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Aug 2013 15:24:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ben.coverston@datastax.com designates 209.85.212.179 as permitted sender) Received: from [209.85.212.179] (HELO mail-wi0-f179.google.com) (209.85.212.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Aug 2013 15:24:21 +0000 Received: by mail-wi0-f179.google.com with SMTP id hr7so1663438wib.0 for ; Mon, 05 Aug 2013 08:24:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=3DSx7+iy1DB9lZWBlj7qx7vW1pdHKV0ItA/dye2DwNY=; b=WASbD74PrypbQ7N5oTUdI96cZFTk/G+DMyZHOesOVes3uasG9ccWsTPUlYq3BizmeZ VBPJ23OiGpW/nwbHr6X0nICj/s9tInZ1nTBoOFeG6isiXt+kNSu/uBqlhDGz+ESF+Ou7 JuyAVSWSoaS22HLmNqG2JL/xu0WwCYyxK9/4kl2CGBgabHG+vMCchi17HHFm7+4g5fwV zeA94TDdEZsWi/u84iFzIMPeqKfawoB1ZF3yBUNYv8KDF6OHZtd0yWxHQXxgnZ6JsUms /OG9/DZ0mVTlZr7qFGWvy2HUVesTnfK9c9zkEYSDNvuKH9zzzjZStNiBZVHB3ktOMbix nSOA== X-Received: by 10.194.243.226 with SMTP id xb2mr13830719wjc.67.1375716240403; Mon, 05 Aug 2013 08:24:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.227.179.1 with HTTP; Mon, 5 Aug 2013 08:23:40 -0700 (PDT) In-Reply-To: References: From: Ben Coverston Date: Mon, 5 Aug 2013 09:23:40 -0600 Message-ID: Subject: Re: org.apache.cassandra.io.sstable.CorruptSSTableException To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e01493538d240fe04e334e519 X-Gm-Message-State: ALoCoQkrHuStd2CHC8sCKZwgDf7z1cR9/7QiE6JgdI3Iiagb9Bzut8R+63b42IrXu4GU8ThmU2Og X-Virus-Checked: Checked by ClamAV on apache.org --089e01493538d240fe04e334e519 Content-Type: text/plain; charset=ISO-8859-1 Also check your system log for IO Errors. Scrub may eliminate the error, but even if it does work you should still run repair. This type of corruption usually happens because of a failed or failing disk/memory. On Mon, Aug 5, 2013 at 8:44 AM, Jason Wee wrote: > you can try nodetool scrub. if it does not work, try repair then cleanup. > had this issue a few weeks back but our version is 1.0.x > > > On Mon, Aug 5, 2013 at 8:12 AM, Keith Wright wrote: > >> Re-sending hoping to get some help. Any ideas would be much appreciated! >> >> From: Keith Wright >> Date: Friday, August 2, 2013 3:01 PM >> To: "user@cassandra.apache.org" >> Subject: org.apache.cassandra.io.sstable.CorruptSSTableException >> >> Hi all, >> >> We just added a node to our cluster (1.2.4 Vnodes) and they appear to >> be running well exception I see that the new node is not making any >> progress compacting one of the CF. The exception below is generated. My >> assumption is that the only way to handle this is to stop the node, delete >> the file in question, restart, and run repair. >> >> Thoughts? >> >> org.apache.cassandra.io.sstable.CorruptSSTableException: >> java.io.IOException: dataSize of 1249463589142530 starting at 5604968 would >> be larger than file >> /data/3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.db >> length 14017479 >> at >> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:168) >> at >> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:83) >> at >> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:69) >> at >> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177) >> at >> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:152) >> at >> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:139) >> at >> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:36) >> at >> org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer$1.runMayThrow(ParallelCompactionIterable.java:288) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> at java.lang.Thread.run(Thread.java:722) >> Caused by: java.io.IOException: dataSize of 1249463589142530 starting at >> 5604968 would be larger than file >> /data/3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.db >> length 14017479 >> at >> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:123) >> ... 9 more >> >> > -- Ben Coverston DataStax -- The Apache Cassandra Company --089e01493538d240fe04e334e519 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Also check your system log for IO Errors. Scrub may elimin= ate the error, but even if it does work you should still run repair. This t= ype of corruption usually happens because of a failed or failing disk/memor= y.


On Mon, Aug 5= , 2013 at 8:44 AM, Jason Wee <peichieh@gmail.com> wrote:
you can try nodetool scrub. if it does not work, try repai= r then cleanup. had this issue a few weeks back but our version is 1.0.x

<= br>
On Mon, Aug 5, 2013 at 8:12 AM, Keith Wright <kwright@nanigans.com= > wrote:
Re-sending hoping to get some he= lp. =A0Any ideas would be much appreciated!

From: Keith Wright <kwright@nanigans.com>=
Date: Friday, August 2, 2013 3= :01 PM
To: "user@cassandra.apache.org" = <user@cas= sandra.apache.org>
Subject: org.apache.cassandra.io.s= stable.CorruptSSTableException

Hi all,

=A0 =A0We just added a node to our cluster= (1.2.4 Vnodes) and they appear to be running well exception I see that the= new node is not making any progress compacting one of the CF. =A0The excep= tion below is generated. =A0My assumption is that the only way to handle this is to stop the node, delete the file in question, restart, and run re= pair.

Thoughts?

org.apach= e.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSi= ze of 1249463589142530 starting at 5604968 would be larger than file /data/= 3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.db length= 14017479
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.sstable.SSTableIdentityIter= ator.<init>(SSTableIdentityIterator.java:168)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>= (SSTableIdentityIterator.java:83)
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.sstable.SSTableIdentityIter= ator.<init>(SSTableIdentityIterator.java:69)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.n= ext(SSTableScanner.java:177)
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.sstable.SSTableScanner$KeyS= canningIterator.next(SSTableScanner.java:152)
=A0 =A0 =A0 =A0 at = org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:139= )
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTa= bleScanner.java:36)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.co= mpaction.ParallelCompactionIterable$Deserializer$1.runMayThrow(ParallelComp= actionIterable.java:288)
=A0 =A0 =A0 =A0 at org.apache.cassandra.utils.WrappedRunnable.run(Wrap= pedRunnable.java:28)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(Thre= ad.java:722)
Caused by: java.io.IOException: dataSize of 12494635= 89142530 starting at 5604968 would be larger than file /data/3/cassandra/da= ta/users/global_user/users-global_user-ib-1550-Data.db length 14017479
=A0 =A0 =A0 =A0 at org.apache.cassandra.io.sstable.SSTableIdentityIter= ator.<init>(SSTableIdentityIterator.java:123)
=A0 =A0 =A0 = =A0 ... 9 more





--
= Ben Coverston
DataStax -- The Apache Cassandra Company
--089e01493538d240fe04e334e519--