Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: 132.72.126.41 is neither permitted nor
 denied by domain of lolitushka@gmail.com)
Message-ID: <4FDA25DC.6060502@gmail.com>
Date: Thu, 14 Jun 2012 20:56:44 +0300
From: Piavlo <lolitushka@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:8.0) Gecko/20111124 Thunderbird/8.0
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Urgent - IllegalArgumentException during compaction and memtable
 flush
References: <4FD983FB.9040901@gmail.com>
 <CAKkz8Q2+w9wFG8ynQXAW9VmbrC5P8oBCCsiWiAXCCuYBRZXz-g@mail.gmail.com>
 <4FD9B635.5020401@gmail.com>
 <CAKkz8Q3GZAYYfTTjQA-0=F=i0XSX6rKMo1LoOL_-nG5tTbWB8A@mail.gmail.com>
 <4FD9EBC1.30409@gmail.com>
 <CAKkz8Q1+jyciPkAPj8auR3uSU2TQF83r328ZBge2j7Ba2QvWNg@mail.gmail.com>
 <4FDA008A.6040404@gmail.com>
In-Reply-To: <4FDA008A.6040404@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


I was able to figure out that 353339332d3134363533393931 is the row key
while no idea what is 92668395684826132216160944211592988451 part?

sstable2json also fails with validation error on this row key

now since I have lost data for this row -  how do I find out that was 
the root cause?

thanks

On 06/14/2012 06:17 PM, Piavlo wrote:
> Ok i've run scrub on the 3 nodes and the problematic row
> Error validating row 
> DecoratedKey(92668395684826132216160944211592988451, 
> 353339332d3134363533393931)
>
> The full message is
>
>  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041 
> CompactionManager.java (line 582) Non-fatal error reading row 
> (stacktrace follows)
> java.io.IOError: java.io.IOException: Error validating row 
> DecoratedKey(92668395684826132216160944211592988451, 
> 353339332d3134363533393931)
>         at 
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
>         at 
> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97)
>         at 
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
>         at 
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:143)
>         at 
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:566)
>         at 
> org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:473)
>         at 
> org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:64)
>         at 
> org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:213)
>         at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
>         at 
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Error validating row 
> DecoratedKey(92668395684826132216160944211592988451, 
> 353339332d3134363533393931)
>         at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:241)
>         at 
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110)
>         ... 13 more
> Caused by: org.apache.cassandra.db.marshal.MarshalException: Not 
> enough bytes to read value of component 1
>         at 
> org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:240)
>         at org.apache.cassandra.db.Column.validateName(Column.java:273)
>         at org.apache.cassandra.db.Column.validateFields(Column.java:278)
>         at 
> org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:372)
>         at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:237)
>         ... 14 more
>  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,085 
> CompactionManager.java (line 624) Row at 4047368880 is unreadable; 
> skipping to next
>
>
> This happened on several sstables on on each of the nodes - meaning it 
> was mutated several times
>
> dsc2b:
>            
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-450-Data.db  at 
> 4244390041
>            
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-452-Data.db  at 
> 9366462649
>
> dsc2c:
>            
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-413-Data.db  at 
> 4047368880
>            
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-481-Data.db  at 
> 3598063925
>
> dsc1a:
>           
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-883-Data.db  at 
> 271195463
>           
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-733-Data.db  at 
> 1089815977
>           
> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-895-Data.db  at 
> 312147765
>
> How do I use this info to find out what is actually wrong with this 
> column, as I highly want to prevent it from being written again :)
>
> Thanks
>
> On 06/14/2012 05:02 PM, Sylvain Lebresne wrote:
>>> Is there way to make cassandra throw away the offending column?
>> Running scrub should allow to get read of the row containing the
>> problematic column. Unfortunately it will discard the whole row, not
>> just the column.
>> However, since scrub takes a snapshot anyway (and should tell you
>> which sstable had the offending column), you could do that and keep
>> the bad sstable around to further investigate.
>>
>>> maybe sstable2json can somehow help?
>> It's worth a try but there is a good change sstable2json will throw an
>> error too. It may have written enough at that point for you to find
>> which row is wrong though.
>>
>> -- 
>> Sylvain
>