cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Getting DigestMismatchExceptions despite setting read repair chances to zero
Date Fri, 27 Oct 2017 14:53:38 GMT


> On Oct 27, 2017, at 3:08 AM, Artur Siekielski <artc@vhex.net> wrote:
> 
> I noticed that the DecoratedKey printed in the stack trace can be for a different table.
The arguments are a token and a partition key and they can be the same for multiple tables.
Is there a way to know for which table the DigestMismatchException happens?
> 

No, the read repair stats we provide are not per table, so if it’s not in the log, it’s
not apparent. Feel free to open a jira to ask for it to be added to the log message.

> Can the AsyncRepairRunner be triggered if read and writes for all other tables are done
with CL=LOCAL_QUORUM (RF=3)? I assumed in that case async read repair is not done even if
dclocal_read_repair_chance > 0. Could it be that the async repair runs for that case and
it's executed faster than the background syncing to meet RF=3?
> 

Very likely. Async repair runner can be triggered if either (dclocal_)read_repair_chance is
> 0. If you write at local_quorum, reads can definitely race (and even in that case, some
writes can be dropped by load shedding or missed during network hiccups or GC pauses). 



> 
>> On 10/26/2017 12:19 PM, Artur Siekielski wrote:
>> Hi,
>> 
>> we have one table for which reads and writes are done with CL=ONE. The table contains
counters. We wanted to disable async read repair for the table (to lessen cluster load and
to avoid DigestMismatchExceptions in debug.log). After altering the table with read_repair_chance=0,
dclocal_read_repair_chance=0, the Digest exceptions still happen:
>> 
>> 
>> DEBUG [ReadRepairStage:92] 2017-10-26 10:00:02,798 ReadCallback.java:242 - Digest
mismatch:
>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(5238932067721150894,
7da6f64695d74899a91bd691321de534) (33f950054869a91d1ea225eae342499a vs 70d054183b9b001de5f71139aa65b8d9)
>>         at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
~[apache-cassandra-3.11.1.jar:3.11.1]
>>         at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
~[apache-cassandra-3.11.1.jar:3.11.1]
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_141]
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_141]
>>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
>>         at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
>> 
>> 
>> I have verified that the DecoratedKey arguments are for a row from the altered table
(by checking token() of the partition key).
>> 
>> Shouldn't the settings disable AsyncRepairRunner for the table?
>> 
>> Is there an option to disable async read repair globally, or each table must be altered?
>> 
>> Cassandra version: 3.11.1
>> 
>> 
>> Thanks,
>> Artur
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message