Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BB98310F04 for ; Mon, 5 Aug 2013 00:13:54 +0000 (UTC) Received: (qmail 60146 invoked by uid 500); 5 Aug 2013 00:13:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 60124 invoked by uid 500); 5 Aug 2013 00:13:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 60116 invoked by uid 99); 5 Aug 2013 00:13:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Aug 2013 00:13:52 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kwright@nanigans.com designates 216.82.243.203 as permitted sender) Received: from [216.82.243.203] (HELO mail1.bemta8.messagelabs.com) (216.82.243.203) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Aug 2013 00:13:45 +0000 Received: from [216.82.242.99:49409] by server-11.bemta-8.messagelabs.com id 47/0B-26861-32EEEF15; Mon, 05 Aug 2013 00:13:23 +0000 X-Env-Sender: kwright@nanigans.com X-Msg-Ref: server-8.tower-131.messagelabs.com!1375661601!5465403!3 X-Originating-IP: [216.166.12.72] X-StarScan-Received: X-StarScan-Version: 6.9.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 29811 invoked from network); 5 Aug 2013 00:13:22 -0000 Received: from out001.collaborationhost.net (HELO out001.collaborationhost.net) (216.166.12.72) by server-8.tower-131.messagelabs.com with RC4-SHA encrypted SMTP; 5 Aug 2013 00:13:22 -0000 Received: from AUSP01VMBX28.collaborationhost.net ([192.168.20.73]) by AUSP01MHUB50.collaborationhost.net ([10.2.10.3]) with mapi; Sun, 4 Aug 2013 19:12:31 -0500 From: Keith Wright To: "user@cassandra.apache.org" Date: Sun, 4 Aug 2013 19:12:35 -0500 Subject: Re: org.apache.cassandra.io.sstable.CorruptSSTableException Thread-Topic: org.apache.cassandra.io.sstable.CorruptSSTableException Thread-Index: Ac6RcHopAuXo6KWjQXqGwM9fohfhYw== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.3.120616 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CE24662413F62kwrightnaniganscom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CE24662413F62kwrightnaniganscom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Re-sending hoping to get some help. Any ideas would be much appreciated! From: Keith Wright > Date: Friday, August 2, 2013 3:01 PM To: "user@cassandra.apache.org" > Subject: org.apache.cassandra.io.sstable.CorruptSSTableException Hi all, We just added a node to our cluster (1.2.4 Vnodes) and they appear to be= running well exception I see that the new node is not making any progress = compacting one of the CF. The exception below is generated. My assumption= is that the only way to handle this is to stop the node, delete the file i= n question, restart, and run repair. Thoughts? org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOExceptio= n: dataSize of 1249463589142530 starting at 5604968 would be larger than fi= le /data/3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.= db length 14017479 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(S= STableIdentityIterator.java:168) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(S= STableIdentityIterator.java:83) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(S= STableIdentityIterator.java:69) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterat= or.next(SSTableScanner.java:177) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterat= or.next(SSTableScanner.java:152) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScann= er.java:139) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScann= er.java:36) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$De= serializer$1.runMayThrow(ParallelCompactionIterable.java:288) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:28) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: dataSize of 1249463589142530 starting at 56= 04968 would be larger than file /data/3/cassandra/data/users/global_user/us= ers-global_user-ib-1550-Data.db length 14017479 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(S= STableIdentityIterator.java:123) ... 9 more --_000_CE24662413F62kwrightnaniganscom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Re-sending hoping to get= some help.  Any ideas would be much appreciated!

=
From: Keith Wright <kwright@nanigans.com>
Date: Friday, August 2, 2013 3:01 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: org.apache.cassandra.io.sstable.CorruptSSTabl= eException

H= i all,

   We just added a node to our cl= uster (1.2.4 Vnodes) and they appear to be running well exception I see tha= t the new node is not making any progress compacting one of the CF.  T= he exception below is generated.  My assumption is that the only way t= o handle this is to stop the node, delete the file in question, restart, and run re= pair.

Thoughts?

org.apach= e.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSi= ze of 1249463589142530 starting at 5604968 would be larger than file /data/= 3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.db length= 14017479
        at org.apache.cassandra.io.= sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:1= 68)
        at org.apache.cassandra.io.sstabl= e.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:83)
        at org.apache.cassandra.io.sstable.SSTab= leIdentityIterator.<init>(SSTableIdentityIterator.java:69)
=         at org.apache.cassandra.io.sstable.SSTableScann= er$KeyScanningIterator.next(SSTableScanner.java:177)
   = ;     at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanni= ngIterator.next(SSTableScanner.java:152)
      &nb= sp; at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.j= ava:139)
        at org.apache.cassandra.io.s= stable.SSTableScanner.next(SSTableScanner.java:36)
    =     at org.apache.cassandra.db.compaction.ParallelCompactionItera= ble$Deserializer$1.runMayThrow(ParallelCompactionIterable.java:288)
        at org.apache.cassandra.utils.WrappedRunnabl= e.run(WrappedRunnable.java:28)
        at jav= a.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException= : dataSize of 1249463589142530 starting at 5604968 would be larger than fil= e /data/3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.d= b length 14017479
        at org.apache.cassa= ndra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterato= r.java:123)
        ... 9 more

 --_000_CE24662413F62kwrightnaniganscom_--