cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Tunnicliffe (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-8479) Timeout Exception on Node Failure in Remote Data Center
Date Thu, 15 Jan 2015 15:54:35 GMT


Sam Tunnicliffe commented on CASSANDRA-8479:

1. read_repair_chance=1 means that on every request, digest requests will be sent to all nodes,
local and remote so this is almost certainly not what you want. As you're reading and writing
at LOCAL_QUORUM, you don't need to worry about read repair. A full read will be done by 1
node in the local dc, and a digest request sent to 1 other local node. This is enough to provide
the consistency you're looking for, as should the digest not match the data a full read will
be done on both nodes and the data resolved according to timestamp. Since you wrote at LOCAL_QUORUM,
at least one of those nodes has the latest value.

You should note that this is not exactly "strong consistency" as you describe it - that would
only be the case with a single client performing both reads and writes & with no failures.
But terminology aside, I believe you can just set read repair chance to 0.

2. What's more relevant than the simple rate is the workload and the delay between writing
a row and reading it back. It's certainly possibly for a write to be ack'd by 2 nodes, satisfying
LOCAL_QUORUM but not yet be processed by the third node. If a read request targets that third
node, either for the full data or a digest you'll get a mismatch.

3. Yes, read_repair_chance=1 means perform a global read repair for 100% of read requests,
which is usually overkill (see CASSANDRA-7320). The logs don't show which node(s) returned
a non-matching digests, only that the coordinator received 3 responses. Those are likely to
have been from the nodes in the same dc, and your experiment with dclocal_read_repair_chance
seems to bear that out, but it isn't guaranteed.

4. Whether read repair happens is only determined by read_repair_chance/dclocal_read_repair_chance
in 2.0, so CL isn't relevant (and has been removed from the documentation - the version you
linked is for Cassandra 1.1).

Note, this all disregards any additional data reads being performed for [Rapid Read Protection|]
because from the logs it seems as though you have that set to NONE for this table.

> Timeout Exception on Node Failure in Remote Data Center
> -------------------------------------------------------
>                 Key: CASSANDRA-8479
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API, Core, Tools
>         Environment: Unix, Cassandra 2.0.11
>            Reporter: Amit Singh Chowdhery
>            Assignee: Sam Tunnicliffe
>            Priority: Minor
>         Attachments:
> Issue Faced :
> We have a Geo-red setup with 2 Data centers having 3 nodes each. When we bring down a
single Cassandra node down in DC2 by kill -9 <Cassandra-pid>, reads fail on DC1 with
TimedOutException for a brief amount of time (15-20 sec~).
> Reference :
> Already a ticket has been opened/resolved and link is provided below :
> Activity Done as per Resolution Provided :
> Upgraded to Cassandra 2.0.11 .
> We have two 3 node clusters in two different DCs and if one or more of the nodes go down
in one Data Center , ~5-10% traffic failure is observed on the other.
> RF=3

This message was sent by Atlassian JIRA

View raw message