cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksey Yeschenko (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-6887) LOCAL_ONE read repair only does local repair, in spite of global digest queries
Date Thu, 29 May 2014 13:19:02 GMT


Aleksey Yeschenko commented on CASSANDRA-6887:

[~slebresne] I would agree with you entirely had this comment been made before we had LOCAL_*
CLs and eager read retries.

Relying on on dc/global read repair chance now, however, is not the most logical/expected
behavior and violates the principle of least surprise.

Currently, without the attached patch:
1. LOCAL_* requests will potentially use a replica from another DC for an eager retry - and
this behavior can *not* be disabled via read repair chance tuning
2. All three of RRD.NONE/GLOBAL/DC_LOCAL can cause a request go to a different DC for LOCAL_*
CL queries, depending on the # of live replicas in the local DC - CL#filterForQuery() is too
late a point to filter the replicas, it must be done before that.
3. A user might *not* want to have read_repair_chance set to 0 entirely, and also use both
LOCAL_* and regular consistency levels - on per-query basis.

So I still think that LOCAL_* CLs should not allow any aspect of the query to involve a non-local
DC - that's arguably the whole point of LOCAL_* CLs, and also, arguably, the expected/least
surprising behavior.

Actually, after spending a bit more time in the code, I'd say that LOCAL_* CLs are flat out
broken/not properly implemented for read requests, with the sole exception of CL#assureSufficientLiveNodes()
that does work properly.

> LOCAL_ONE read repair only does local repair, in spite of global digest queries
> -------------------------------------------------------------------------------
>                 Key: CASSANDRA-6887
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.0.6, x86-64 ubuntu precise
>            Reporter: Duncan Sands
>            Assignee: Aleksey Yeschenko
>             Fix For: 2.0.9, 2.1 rc1
>         Attachments: 6887-2.0.txt
> I have a cluster spanning two data centres.  Almost all of the writing (and a lot of
reading) is done in DC1.  DC2 is used for running the occasional analytics query.  Reads in
both data centres use LOCAL_ONE.  Read repair settings are set to the defaults on all column
> I had a long network outage between the data centres; it lasted longer than the hints
window, so after it was over DC2 didn't have the latest information.  Even after reading data
many many times in DC2, the returned data was still out of date: read repair was not correcting
> I then investigated using cqlsh in DC2, with tracing on.
> What I saw was:
>   - with consistency ONE, after about 10 read requests a digest request would be sent
to many nodes (spanning both data centres), and the data in DC2 would be repaired.
>  - with consistency LOCAL_ONE, after about 10 read requests a digest request would be
sent to many nodes (spanning both data centres), but the data in DC2 would not be repaired.
 This is in spite of digest requests being sent to DC1, as shown by the tracing.
> So it looks like digest requests are being sent to both data centres, but replies from
outside the local data centre are ignored when using LOCAL_ONE.
> The same data is being queried all the time in DC1 with consistency LOCAL_ONE, but this
didn't result in the data in DC2 being read repaired either.  This is a slightly different
case to what I described above: in that case the local node was out of date and the remote
node had the latest data, while here it is the other way round.
> It could be argued that you don't want cross data centre read repair when using LOCAL_ONE.
 But then why bother sending cross data centre digest requests?  And if only doing local read
repair is how it is supposed to work then it would be good to document this somewhere.

This message was sent by Atlassian JIRA

View raw message