cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piavlo <lolitus...@gmail.com>
Subject Re: Urgent - IllegalArgumentException during compaction and memtable flush
Date Thu, 14 Jun 2012 15:17:30 GMT
Ok i've run scrub on the 3 nodes and the problematic row
Error validating row 
DecoratedKey(92668395684826132216160944211592988451, 
353339332d3134363533393931)

The full message is

  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041 
CompactionManager.java (line 582) Non-fatal error reading row 
(stacktrace follows)
java.io.IOError: java.io.IOException: Error validating row 
DecoratedKey(92668395684826132216160944211592988451, 
353339332d3134363533393931)
         at 
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
         at 
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97)
         at 
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
         at 
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:143)
         at 
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:566)
         at 
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:473)
         at 
org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:64)
         at 
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:213)
         at 
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error validating row 
DecoratedKey(92668395684826132216160944211592988451, 
353339332d3134363533393931)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:241)
         at 
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110)
         ... 13 more
Caused by: org.apache.cassandra.db.marshal.MarshalException: Not enough 
bytes to read value of component 1
         at 
org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:240)
         at org.apache.cassandra.db.Column.validateName(Column.java:273)
         at org.apache.cassandra.db.Column.validateFields(Column.java:278)
         at 
org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:372)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:237)
         ... 14 more
  WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,085 
CompactionManager.java (line 624) Row at 4047368880 is unreadable; 
skipping to next


This happened on several sstables on on each of the nodes - meaning it 
was mutated several times

dsc2b:
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-450-Data.db  at 
4244390041
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-452-Data.db  at 
9366462649

dsc2c:
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-413-Data.db  at 
4047368880
            
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-481-Data.db  at 
3598063925

dsc1a:
           
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-883-Data.db  at 
271195463
           
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-733-Data.db  at 
1089815977
           
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-895-Data.db  at 
312147765

How do I use this info to find out what is actually wrong with this 
column, as I highly want to prevent it from being written again :)

Thanks

On 06/14/2012 05:02 PM, Sylvain Lebresne wrote:
>> Is there way to make cassandra throw away the offending column?
> Running scrub should allow to get read of the row containing the
> problematic column. Unfortunately it will discard the whole row, not
> just the column.
> However, since scrub takes a snapshot anyway (and should tell you
> which sstable had the offending column), you could do that and keep
> the bad sstable around to further investigate.
>
>> maybe sstable2json can somehow help?
> It's worth a try but there is a good change sstable2json will throw an
> error too. It may have written enough at that point for you to find
> which row is wrong though.
>
> --
> Sylvain


Mime
View raw message