cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom van den Berge (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9753) LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch
Date Wed, 09 Sep 2015 18:49:47 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737396#comment-14737396
] 

Tom van den Berge commented on CASSANDRA-9753:
----------------------------------------------

I was setting up a new DC, and noticed that other DCs were sending read queries to this DC,
even though clients did not connect to the new DC, and all queries were using LOCAL_* consistency.
This bug proved to be the cause of this problem.

I was able to work around it by disabling speculative_retry on all tables:
  alter table <table> with speculative_retry='NONE';

> LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-9753
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9753
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Richard Low
>
> When there is a digest mismatch during the initial read, a data read request is sent
to all replicas involved in the initial read. This can be more than the initial blockFor if
read repair was done and if speculative retry kicked in. E.g. for RF 3 in two DCs, the number
of reads could be 4: 2 for LOCAL_QUORUM, 1 for read repair and 1 for speculative read if one
replica was slow. If there is then a digest mismatch, Cassandra will issue the data read to
all 4 and set blockFor=4. Now the read query is blocked on cross-DC latency. The digest mismatch
read blockFor should be capped at RF for the local DC when using CL.LOCAL_*.
> You can reproduce this behaviour by creating a keyspace with NetworkTopologyStrategy,
RF 3 per DC, dc_local_read_repair=1.0 and ALWAYS for speculative read. If you force a digest
mismatch (e.g. by deleting a replicas SSTables and restarting) you can see in tracing that
it is blocking for 4 responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message