incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Brosius <dbros...@mebigfatguy.com>
Subject Re: Urgent - IllegalArgumentException during compaction and memtable flush
Date Thu, 14 Jun 2012 18:55:56 GMT
One of the column names on the row with key 353339332d3134363533393931 
failed to validate with the validator for the column.

If you really are after what column is problematic, and are able to 
build and run cassandra, you can add debugging info to Column.java

     protected void validateName(CFMetaData metadata) throws 
MarshalException
     {
*try {*
         AbstractType<?> nameValidator = metadata.cfType == 
ColumnFamilyType.Super ? metadata.subcolumnComparator : metadata.comparator;

         nameValidator.validate(name());
*} catch (MarshalException me) {
     throw new MarshalException("Failed validating name: " + 
ByteBufferUtil.bytesToHex(name()), me);
}*
     }

btw, the 92668395684826132216160944211592988451 is just the key's token.



On 06/14/2012 01:56 PM, Piavlo wrote:
>
> I was able to figure out that 353339332d3134363533393931 is the row key
> while no idea what is 92668395684826132216160944211592988451 part?
>
> sstable2json also fails with validation error on this row key
>
> now since I have lost data for this row -  how do I find out that was 
> the root cause?
>
> thanks     protected void validateName(CFMetaData metadata) throws 
> MarshalException
>     {
>         AbstractType<?> nameValidator = metadata.cfType == 
> ColumnFamilyType.Super ? metadata.subcolumnComparator : 
> metadata.comparator;
>         nameValidator.validate(name());
>     }
>
> On 06/14/2012 06:17 PM, Piavlo wrote:
>> Ok i've run scrub on the 3 nodes and the problematic row
>> Error validating row 
>> DecoratedKey(92668395684826132216160944211592988451, 
>> 353339332d3134363533393931)
>>
>> The full message is
>>
>>  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041 
>> CompactionManager.java (line 582) Non-fatal error reading row 
>> (stacktrace follows)
>> java.io.IOError: java.io.IOException: Error validating row 
>> DecoratedKey(92668395684826132216160944211592988451, 
>> 353339332d3134363533393931)
>>         at 
>> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
>>         at 
>> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:143)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:566)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:473)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:64)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:213)
>>         at 
>> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
>>         at 
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: Error validating row 
>> DecoratedKey(92668395684826132216160944211592988451, 
>> 353339332d3134363533393931)
>>         at 
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:241)
>>         at 
>> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110)
>>         ... 13 more
>> Caused by: org.apache.cassandra.db.marshal.MarshalException: Not 
>> enough bytes to read value of component 1
>>         at 
>> org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:240)
>>         at org.apache.cassandra.db.Column.validateName(Column.java:273)
>>         at 
>> org.apache.cassandra.db.Column.validateFields(Column.java:278)
>>         at 
>> org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:372)
>>         at 
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:237)
>>         ... 14 more
>>  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,085 
>> CompactionManager.java (line 624) Row at 4047368880 is unreadable; 
>> skipping to next
>>
>>
>> This happened on several sstables on on each of the nodes - meaning 
>> it was mutated several times
>>
>> dsc2b:
>>            
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-450-Data.db  at 
>> 4244390041
>>            
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-452-Data.db  at 
>> 9366462649
>>
>> dsc2c:
>>            
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-413-Data.db  at 
>> 4047368880
>>            
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-481-Data.db  at 
>> 3598063925
>>
>> dsc1a:
>>           
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-883-Data.db  at 
>> 271195463
>>           
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-733-Data.db  at 
>> 1089815977
>>           
>> /var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-895-Data.db  at 
>> 312147765
>>
>> How do I use this info to find out what is actually wrong with this 
>> column, as I highly want to prevent it from being written again :)
>>
>> Thanks
>>
>> On 06/14/2012 05:02 PM, Sylvain Lebresne wrote:
>>>> Is there way to make cassandra throw away the offending column?
>>> Running scrub should allow to get read of the row containing the
>>> problematic column. Unfortunately it will discard the whole row, not
>>> just the column.
>>> However, since scrub takes a snapshot anyway (and should tell you
>>> which sstable had the offending column), you could do that and keep
>>> the bad sstable around to further investigate.
>>>
>>>> maybe sstable2json can somehow help?
>>> It's worth a try but there is a good change sstable2json will throw an
>>> error too. It may have written enough at that point for you to find
>>> which row is wrong though.
>>>
>>> -- 
>>> Sylvain
>>
>
>


Mime
View raw message