incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Lowe <richard.l...@arkivum.com>
Subject RE: Question about consistency levels
Date Sun, 10 Nov 2013 20:13:26 GMT
We're using Cassandra 1.1 with Hector 1.1 library. We've found that reducing the CL when an
exception occurs is useful as it's usually easier to deal with things not being consistent
for a few seconds than the database read/write not succeeding at all.

We have multiple DCs and use NetworkTopologyStrategy to strictly govern where data is replicated.
Because of this, LOCAL_QUORUM isn't good enough: if a write occurs in DC A then nodes in DC
B won't necessarily have the data before a read in DC B wants the (maybe updated) data. 

We therefore use ALL, EACH_QUORUM, QUORUM, LOCAL_QUORUM, ONE, ANY for write CL, falling back
to the next in the list if a CL can't be achieved. With high-consistency writes, we can get
away with lower-consistency reads, so use LOCAL_QUORUM, ONE for read CL, again falling back
to the next if an error occurs. We also find it useful to retry with several attempts at each
CL to account for network faults, which can be common on a congested/slow/unreliable network
(especially WAN) or busy nodes. We only fail the operation if we've exhausted all CL and retries.


This approach works well for us: we issue a warning if a lower CL is used or retries were
required and we've seen only a handful of these on our nodes and then only during very busy
periods. It helps that our use case allows us to occasionally take a couple of extra seconds
to perform the database op if we need to; yours may not, I don't know.

We haven't had any problems with errors from corrupt SSTables but maybe that's because we're
using a different version of Cassandra, a different client and likely have different read/write/delete
usage.

Hope that helps.

-----Original Message-----
From: graham sanderson [mailto:graham@vast.com] 
Sent: 10 November 2013 06:12
To: user@cassandra.apache.org
Subject: Question about consistency levels

I'm trying to be more succinct this time since no answers on my last attempt.

We are currently using 2.0.2 in test (no C* in production yet), and use (LOCAL_)QUORUM CL
on read and writes which guarantees (if successful) that we read latest data.

That said, it is highly likely that (LOCAL_)ONE would return our data since it isn't read
for quite some time after write.

Given that we must do our best to return data, we want to see what options we have when a
quorum read fails (say 2 of 3 replicas go down with 3 replicas - note we have also seen this
issue with bugs related to CF deletion/re-creating during compaction or load causing data
corruption in which case 1 bad node can screw things up)

One option is to fall back to (LOCAL_)ONE if we detect the right exception from (LOCAL_)QUORUM
from the client side, but that obviously degrades your consistency.

That said we ONLY ever do idempotent writes, and NEVER delete. So once again I wonder if there
is a (reasonable) use case for a CL whereby you will accept the first non empty response from
any replica?

Mime
View raw message