cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliott Sims <>
Subject Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Date Sat, 13 Oct 2018 03:16:58 GMT
I'll second that - we had some weird inconsistent reads for a long time
that we finally tracked to a small number of clients with significant clock
skew.  Make very sure all your client (not just C*) machines have
tightly-synced clocks.

On Fri, Oct 12, 2018 at 7:40 PM maitrayee shah <>

> We have seen inconsistent read if the clock on the nodes are not in sync.
> Thank you
> Sent from my iPhone
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad <> wrote:
> Hello,
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
> Now, while reading we're seeing the following cases of inconsistent reads:
>    - One column was added. If a read was done after the column was added
>    (20 secs to 2 minutes after the write), Cassandra returns no data. As if
>    the key doesn't exist. If the application retries, it gets the data.
>    - A few columns exist for a row key. And a new column 'n' was added.
>    Again, a read happens a few minutes after the write. This time, only the
>    latest column 'n' is returned. In this case the app doesn't know that the
>    data is incomplete so it doesn't retry. If we manually retry, we see all
>    the columns.
>    - A few columns exist for a row key. And a new column 'n' is added.
>    When a read happens after the write, all columns but 'n' are returned.
> Here's what we've verified:
>    - Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>    - The replication is within local data center. No remote data center
>    is involved in the read or write.
>    - During the inconsistent reads, none of the nodes are undergoing GC
>    pauses
>    - There are no errors in cassandra logs
>    - Reads always happen after the writes.
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: Replication Factor: 3
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
> Here's how the table definition looks like:
>   key text,
>   sub_key text,
>   value text,
>   PRIMARY KEY ((key), sub_key)
> ) WITH
>   bloom_filter_fp_chance=0.010000 AND
>   caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.100000 AND
>   gc_grace_seconds=864000 AND
>   read_repair_chance=0.000000 AND
>   default_time_to_live=0 AND
>   speculative_retry='ALWAYS' AND
>   memtable_flush_period_in_ms=0 AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={'sstable_compression': 'LZ4Compressor'};
> Please point us in the right direction. Thanks !
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.

View raw message