cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hazel Bobrins (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9715) Secondary index out of sync
Date Thu, 02 Jul 2015 20:38:04 GMT


Hazel Bobrins commented on CASSANDRA-9715:

Updated - sorry seem to have posted blank before.

> Secondary index out of sync
> ---------------------------
>                 Key: CASSANDRA-9715
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: RHEL 6.2 2.6.32-220.13.1.el6.x86_64
>            Reporter: Hazel Bobrins
> On 2.0.15 ( we moved from 2.08 hoping this problem would go away) we am seeing intermittent
issues where a secondary index is getting out of sync.
> Set up is a 6 node cluster with 3 data centers, two nodes in each and with a RF of 2
in each data centre.
> So far I have been unable to reproduce this synthetically but have seen multiple instances
across all nodes within the cluster.
> Data set is very small ~40K keys and >100MB of data. We add maybe 1000 records a day,
delete ~500 and update ~200. Not a very write based system. Reads we can push out to ~2000/sec.
> Writes are done at CL ALL and reads at ONE
> All examples so far have been triggered when a record has been deleted and then other
added with the same index cardinality; I think it has also always been the last record in
the set which was deleted before the addition.
> On a flushed keyspace a sstable2json export of the primary index shows all records correctly,
however, an export of the secondary index is missing the records.
> nodetool rebuild_index does not resolve the problem
> Nether does a compact or repair
> A select on the primary key at CL ALL also has no impact
> However, a select at CL ALL on the secondary index does resolve the problem.
> There is currently a none critical record which is out of the index on one of our nodes.
If another key is added with the same index cardinality it is added to the index correctly.
If this is then removed it once again returns empty.
> We have checked all the obvious OS bits and confirmed our time sync (ntp based).
> At DEBUG level we see nothing obvious wrong when adding/removing keys to the above broken
> Due to the very intermittent nature of this problem is been impossible so far to gather
any DEBUG logs of it failing; we have also been unsuccessful so far in reproducing this in
out QA.
> I know this is not much to go on, if there is anything we can provide to help expand
what might be the issue please let me know and we'll provide it asap.

This message was sent by Atlassian JIRA

View raw message