Also I see
WARN  [ReadRepairStage:341] 2018-12-31 17:57:58,594 DataResolver.java:507 - Encountered an oversized (37264537/16777216) read repair mutation for table

for about several hours (5-7) after cluster restart, next it disappeared.


On Tuesday, January 1, 2019 1:10 PM, Vlad <qa23d-vvd@yahoo.com.INVALID> wrote:


Actually what happened is that Cassandra instances was upgraded to bigger size one by one, with downtime about one-two minutes each. There are logs

INFO  [HintsDispatcher:3] 2018-12-31 11:23:48,305 HintsDispatchExecutor.java:282 - Finished hinted handoff of file d2c7bb82-3d7a-43b2-8791-ef9c7c02b2c1-1546182664318-1.hints to endpoint /10.123.123.123: d2c7bb82-3d7a-43b2-8791-ef9c7c02b2c1

And from now on there is lot of Digest Mismatch exceptions in Cassandra log.

What's going on?




On Tuesday, January 1, 2019 10:18 AM, Vlad <qa23d-vvd@yahoo.com.INVALID> wrote:


Hi All and Happy New Year!!!

This year started with Cassandra 3.11.3 sometimes forces level ALL despite query level LOCAL_QUORUM (actually there is only one DC) and it fails with timeout.

As far as I understand, it can be caused by read repair attempts (we see "DigestMismatch" errors in Cassandra log), but table has no read repair configured:

    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';


Any suggestions?

Thanks.