cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naik, Ninad" <>
Subject Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Date Fri, 12 Oct 2018 20:50:09 GMT

We're seeing inconsistent data while doing reads on cassandra. Here are the details:

It's is a wide column table. The columns can be added my multiple machines, and read by multiple
machines. The time between writes and reads are in minutes, but sometimes can be in seconds.
Writes happen every 2 minutes.

Now, while reading we're seeing the following cases of inconsistent reads:

  *   One column was added. If a read was done after the column was added (20 secs to 2 minutes
after the write), Cassandra returns no data. As if the key doesn't exist. If the application
retries, it gets the data.
  *   A few columns exist for a row key. And a new column 'n' was added. Again, a read happens
a few minutes after the write. This time, only the latest column 'n' is returned. In this
case the app doesn't know that the data is incomplete so it doesn't retry. If we manually
retry, we see all the columns.
  *   A few columns exist for a row key. And a new column 'n' is added. When a read happens
after the write, all columns but 'n' are returned.

Here's what we've verified:

  *   Both writes and reads are using 'LOCAL_QUORUM' consistency level.
  *   The replication is within local data center. No remote data center is involved in the
read or write.
  *   During the inconsistent reads, none of the nodes are undergoing GC pauses
  *   There are no errors in cassandra logs
  *   Reads always happen after the writes.

A few other details: Cassandra version: 2.1.9 DataStax java driver version: Replication
Factor: 3

We don't see this problem in lower environments. We have seen this happen once or twice last
year, but since last few days it's happening quite frequently. On an average 2 inconsistent
reads every minute.

Here's how the table definition looks like:

  key text,
  sub_key text,
  value text,
  PRIMARY KEY ((key), sub_key)
  bloom_filter_fp_chance=0.010000 AND
  caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
  comment='' AND
  dclocal_read_repair_chance=0.100000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.000000 AND
  default_time_to_live=0 AND
  speculative_retry='ALWAYS' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

Please point us in the right direction. Thanks !

The information contained in this e-mail message and any attachments may be privileged and
confidential. If the reader of this message is not the intended recipient or an agent responsible
for delivering it to the intended recipient, you are hereby notified that any review, dissemination,
distribution or copying of this communication is strictly prohibited. If you have received
this communication in error, please notify the sender immediately by replying to this e-mail
and delete the message and any attachments from your computer.

View raw message