incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Re: Token != DecoratedKey assertion
Date Thu, 24 Nov 2011 12:31:18 GMT
I'm seeing this error on a 0.8.x node again. This node did suffer a crash
and the cassandra data is on a raid0 array. The array was remounted
correctly and the xfs filesystem did not report any issues.

Given that RF=3 I have the following question:
0. Can a storage problem cause this?
1. During reads is data from this node ignored from QUORUM when the
assertion is raised?
2. Will the key be read repaired?
3. Can someone confirm what Aaron 'thinks' (scrub vs repair)
4. Can data in my cluster have been corrupted on multiple nodes if my
storage went bad on that node ?

Thanks
On Oct 5, 2011 12:22 PM, "Philippe" <watcherfr@gmail.com> wrote:

> A little feedback,
> I scrubbed on each server and I haven't seen this error again. The load on
> each server eems to be correct.
> nodetool compactionstats shows  boat-load of "Scrub" at 100% on my 3rd
> node but not on the 2 others.
> I left it that way and haven't restart yet.
>
> 2011/9/26 aaron morton <aaron@thelastpickle.com>
>
>> Looks like a mismatch between the key the index says should be at a
>> certain position in the date file and the key that is actually there.
>>
>> I've not checked but scrub *may* fix this this. Try it and see.
>>
>> (repair is for repairing consistency between nodes, scrub fixes local
>> issues with data. )
>>
>> Cheers
>>
>>  -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 26/09/2011, at 12:53 PM, Philippe wrote:
>>
>> Juste did
>> Could there be data corruption or will repairs do this?
>>
>> Thanks
>> Le 25 sept. 2011 15:30, "Jonathan Ellis" <jbellis@gmail.com> a écrit :
>> > Assertion errors are bugs, so that should worry you.
>> >
>> > However, I'd upgrade before filing a ticket. There were a lot of
>> > fixes in 0.8.5.
>> >
>> > On Sun, Sep 25, 2011 at 2:27 AM, Philippe <watcherfr@gmail.com> wrote:
>> >> Hello,
>> >> I've seen a couple of these in my logs, running 0.8.4.
>> >> This is a RF=3, 3-node cluster. 2 nodes including this one are on
>> 0.8.4 and
>> >> one is on 0.8.5
>> >>
>> >> The node is still functionning hours later. Should I be worried ?
>> >>
>> >> Thanks
>> >>
>> >> ERROR [ReadStage:94911] 2011-09-24 22:40:30,043
>> AbstractCassandraDaemon.java
>> >> (line 134) Fatal exception in thread Thread[ReadStage:94911,5,main]
>> >> java.lang.AssertionError:
>> >>
>> DecoratedKey(Token(bytes[0000224ceb80b5fb11e00000848783ceb9bf0000000000000002ffffffffffffff33]),
>> >> 0000224ceb80b5fb11e00000848783ceb9bf0000000000000002ffffffffffffff33)
>> !=
>> >> DecoratedKey(Token(bytes[038453154cb00000005f14]),
>> 038453154cb00000005f14)
>> >> in /var/lib/cassandra/data/XXXXXXXXX/PUBLIC_MONTHLY_20-g-10634-Data.db
>> >>         at
>> >>
>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>> >>         at
>> >>
>> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
>> >>         at
>> >>
>> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169)
>> >>         at org.apache.cassandra.db.Table.getRow(Table.java:385)
>> >>         at
>> >>
>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
>> >>         at
>> >>
>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642)
>> >>         at
>> >>
>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>         at java.lang.Thread.run(Thread.java:662)
>> >> ERROR [ReadStage:94936] 2011-09-24 22:40:30,042
>> AbstractCassandraDaemon.java
>> >> (line 134) Fatal exception in thread Thread[ReadStage:94936,5,main]
>> >> java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) !=
>> >> DecoratedKey(Token(bytes[038453154c900000005f14]),
>> 038453154c900000005f14)
>> >> in /var/lib/cassandra/data/XXXXXXXXX/PUBLIC_MONTHLY_20-g-10634-Data.db
>> >>         at
>> >>
>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>> >>         at
>> >>
>> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
>> >>         at
>> >>
>> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222)
>> >>         at
>> >>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169)
>> >>         at org.apache.cassandra.db.Table.getRow(Table.java:385)
>> >>         at
>> >>
>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
>> >>         at
>> >>
>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642)
>> >>         at
>> >>
>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>         at java.lang.Thread.run(Thread.java:662)
>> >> ERROR [ReadStage:94713] 2011-09-24 22:40:30,041
>> AbstractCassandraDaemon.java
>> >> (line 134) Fatal exception in thread Thread[ReadStage:94713,5,main]
>> >> java.lang.AssertionError:
>> >>
>> DecoratedKey(Token(bytes[7c4831fe0000000000000001ffffffffffffffaa000c6c697665626f782d6265306580000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffb0000c6c697665626f782d6631326380000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffab000b062a27f9e3000000005f13000000]),
>> >>
>> 7c4831fe0000000000000001ffffffffffffffaa000c6c697665626f782d6265306580000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffb0000c6c697665626f782d6631326380000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffab000b062a27f9e3000000005f13000000)
>> >> != DecoratedKey(Token(bytes[038451bfeeb00000005f14]),
>> >> 038451bfeeb00000005f14) in
>> >> /var/lib/cassandra/data/XXXXXXXXX/PUBLIC_MONTHLY_20-g-12305-Data.db
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>>
>>
>>
>

Mime
View raw message