cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Cheng <br...@blockcypher.com>
Subject Re: inconsistent results
Date Wed, 15 Feb 2017 00:01:08 GMT
Change your consistency levels in the cqlsh shell while you query, from ONE
to QUORUM to ALL. If you see your results change that's a consistency
issue. (Assuming these are simple inserts, if there's deletes, potentially
update collections, etc. in the mix then things get a bit more complex.)

To diagnose why the issue exists, a helpful metric are the various dropped
messages metrics from nodetool tpstats. Overloaded clusters will experience
consistency issues as a result of dropped mutations.

It's helpful to think of things in terms of guarantees. If you write with
CL=ONE or LOCAL_ONE, you're getting exactly one guaranteed write. In a
healthy system with tons of excess capacity, you will likely see much
better consistency than that; the hint system will replicate the write to
other nodes, which will perform the write if they can. Since it appears
you're seeing inconsistency at CL=ONE, plus timeouts at CL=QUORUM, it's
quite likely your cluster is not capable of keeping up with the consistency
level you require.

Why your cluster is overloaded is another question entirely, but if you
discover that's the case in my experience the most common cases are
excessive GC due to bad heap settings and data model issues that cause
massive partitions.

On Tue, Feb 14, 2017 at 2:03 PM, Josh England <josh@tgsmc.com> wrote:

> I suspect this is true, but it has proven to be significantly harder to
> track down.  Either cassandra is tickling some bug that nothing else does
> or something strange is going on internally.  On an otherwise quiet system,
> I'd see instant results most of the time intermixed with queries (reads)
> that would timeout and fail.  I agree this needs to be addressed but I'd
> like to understand what is currently going on with my queries.  If it is
> thought to be a consistency problem, how can that be verified?
>
> -JE
>
>
> On Tue, Feb 14, 2017 at 1:46 PM, Jonathan Haddad <jon@jonhaddad.com>
> wrote:
>
>> If you're getting a lot of timeouts you will almost certainly end up with
>> consistency issues. You're going to need to fix the root cause, your
>> cluster instability, or this sort of issue will be commonplace.
>>
>>
>> On Tue, Feb 14, 2017 at 1:43 PM Josh England <josh@tgsmc.com> wrote:
>>
>>> I'll try it the repair.  Using quorum tends to lead to too many timeout
>>> problems though.  :(
>>>
>>> -JE
>>>
>>>
>>> On Tue, Feb 14, 2017 at 1:39 PM, Oskar Kjellin <oskar.kjellin@gmail.com>
>>> wrote:
>>>
>>> Repair might help. But you will end up in this situation again unless
>>> you read/write using quorum (may be local)
>>>
>>> Sent from my iPhone
>>>
>>> On 14 Feb 2017, at 22:37, Josh England <josh@tgsmc.com> wrote:
>>>
>>> All client interactions are from python (python-driver 3.7.1) using
>>> default consistency (LOCAL_ONE I think).  Should I try repairing all nodes
>>> to make sure all data is consistent?
>>>
>>> -JE
>>>
>>>
>>> On Tue, Feb 14, 2017 at 1:32 PM, Oskar Kjellin <oskar.kjellin@gmail.com>
>>> wrote:
>>>
>>> What consistency levels are you using for reads/writes?
>>>
>>> Sent from my iPhone
>>>
>>> > On 14 Feb 2017, at 22:27, Josh England <josh@tgsmc.com> wrote:
>>> >
>>> > I'm running Cassandra 3.9 on CentOS 6.7 in a 6-node cluster.  I've got
>>> a situation where the same query sometimes returns 2 records (correct), and
>>> sometimes only returns 1 record (incorrect).  I've ruled out the
>>> application and the indexing since this is reproducible directly from a
>>> cqlsh shell with a simple select statement.  What is the best way to debug
>>> what is happening here?
>>> >
>>> > -JE
>>> >
>>>
>>>
>>>
>>>
>

Mime
View raw message