Return-Path: 
 <user-return-27011-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F15590D9
	for <apmail-cassandra-user-archive@www.apache.org>;
 Thu, 14 Jun 2012 15:16:58 +0000 (UTC)
Received: (qmail 73840 invoked by uid 500); 14 Jun 2012 15:16:55 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 73801 invoked by uid 500); 14 Jun 2012 15:16:55 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 73793 invoked by uid 99); 14 Jun 2012 15:16:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jun 2012 15:16:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=FSL_RCVD_USER,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 132.72.42.23 is neither permitted nor
 denied by domain of lolitushka@gmail.com)
Received: from [132.72.42.23] (HELO indigo.cs.bgu.ac.il) (132.72.42.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jun 2012 15:16:46 +0000
Received: from [132.72.41.93] (lesbinux [132.72.41.93])
	by indigo.cs.bgu.ac.il (8.13.6/8.13.6) with ESMTP id q5EFUtYf023647
	for <user@cassandra.apache.org>; Thu, 14 Jun 2012 18:30:55 +0300 (IDT)
Message-ID: <4FDA008A.6040404@gmail.com>
Date: Thu, 14 Jun 2012 18:17:30 +0300
From: Piavlo <lolitushka@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Urgent - IllegalArgumentException during compaction and memtable
 flush
References: <4FD983FB.9040901@gmail.com>
 <CAKkz8Q2+w9wFG8ynQXAW9VmbrC5P8oBCCsiWiAXCCuYBRZXz-g@mail.gmail.com>
 <4FD9B635.5020401@gmail.com>
 <CAKkz8Q3GZAYYfTTjQA-0=F=i0XSX6rKMo1LoOL_-nG5tTbWB8A@mail.gmail.com>
 <4FD9EBC1.30409@gmail.com>
 <CAKkz8Q1+jyciPkAPj8auR3uSU2TQF83r328ZBge2j7Ba2QvWNg@mail.gmail.com>
In-Reply-To: 
 <CAKkz8Q1+jyciPkAPj8auR3uSU2TQF83r328ZBge2j7Ba2QvWNg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on gandalf
X-Old-Spam-Status: No, score=-1.8 required=6.5 tests=ALL_TRUSTED autolearn=no
	version=3.2.3

Ok i've run scrub on the 3 nodes and the problematic row
Error validating row 
DecoratedKey(92668395684826132216160944211592988451, 
353339332d3134363533393931)

The full message is

  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041 
CompactionManager.java (line 582) Non-fatal error reading row 
(stacktrace follows)
java.io.IOError: java.io.IOException: Error validating row 
DecoratedKey(92668395684826132216160944211592988451, 
353339332d3134363533393931)
         at 
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
         at 
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97)
         at 
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
         at 
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:143)
         at 
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:566)
         at 
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:473)
         at 
org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:64)
         at 
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:213)
         at 
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error validating row 
DecoratedKey(92668395684826132216160944211592988451, 
353339332d3134363533393931)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:241)
         at 
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110)
         ... 13 more
Caused by: org.apache.cassandra.db.marshal.MarshalException: Not enough 
bytes to read value of component 1
         at 
org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:240)
         at org.apache.cassandra.db.Column.validateName(Column.java:273)
         at org.apache.cassandra.db.Column.validateFields(Column.java:278)
         at 
org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:372)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:237)
         ... 14 more
  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,085 
CompactionManager.java (line 624) Row at 4047368880 is unreadable; 
skipping to next


This happened on several sstables on on each of the nodes - meaning it 
was mutated several times

dsc2b:
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-450-Data.db  at 
4244390041
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-452-Data.db  at 
9366462649

dsc2c:
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-413-Data.db  at 
4047368880
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-481-Data.db  at 
3598063925

dsc1a:
           
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-883-Data.db  at 
271195463
           
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-733-Data.db  at 
1089815977
           
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-895-Data.db  at 
312147765

How do I use this info to find out what is actually wrong with this 
column, as I highly want to prevent it from being written again :)

Thanks

On 06/14/2012 05:02 PM, Sylvain Lebresne wrote:
>> Is there way to make cassandra throw away the offending column?
> Running scrub should allow to get read of the row containing the
> problematic column. Unfortunately it will discard the whole row, not
> just the column.
> However, since scrub takes a snapshot anyway (and should tell you
> which sstable had the offending column), you could do that and keep
> the bad sstable around to further investigate.
>
>> maybe sstable2json can somehow help?
> It's worth a try but there is a good change sstable2json will throw an
> error too. It may have written enough at that point for you to find
> which row is wrong though.
>
> --
> Sylvain