Ok i've run scrub on the 3 nodes and the problematic row
Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
The full message is
WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041
CompactionManager.java (line 582) Non-fatal error reading row
(stacktrace follows)
java.io.IOError: java.io.IOException: Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:143)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:566)
at
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:473)
at
org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:64)
at
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:213)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:241)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110)
... 13 more
Caused by: org.apache.cassandra.db.marshal.MarshalException: Not enough
bytes to read value of component 1
at
org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:240)
at org.apache.cassandra.db.Column.validateName(Column.java:273)
at org.apache.cassandra.db.Column.validateFields(Column.java:278)
at
org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:372)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:237)
... 14 more
WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,085
CompactionManager.java (line 624) Row at 4047368880 is unreadable;
skipping to next
This happened on several sstables on on each of the nodes - meaning it
was mutated several times
dsc2b:
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-450-Data.db at
4244390041
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-452-Data.db at
9366462649
dsc2c:
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-413-Data.db at
4047368880
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-481-Data.db at
3598063925
dsc1a:
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-883-Data.db at
271195463
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-733-Data.db at
1089815977
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-895-Data.db at
312147765
How do I use this info to find out what is actually wrong with this
column, as I highly want to prevent it from being written again :)
Thanks
On 06/14/2012 05:02 PM, Sylvain Lebresne wrote:
>> Is there way to make cassandra throw away the offending column?
> Running scrub should allow to get read of the row containing the
> problematic column. Unfortunately it will discard the whole row, not
> just the column.
> However, since scrub takes a snapshot anyway (and should tell you
> which sstable had the offending column), you could do that and keep
> the bad sstable around to further investigate.
>
>> maybe sstable2json can somehow help?
> It's worth a try but there is a good change sstable2json will throw an
> error too. It may have written enough at that point for you to find
> which row is wrong though.
>
> --
> Sylvain
|