There is now a reproducible test case.
I had similar issue (reported many times here, there's also a JIRA issue, but people reporting this problem were unable to reproduce it).
What I can say is that for me the solution was to run major compaction on the index CF via JMX. To be clear - we're not talking about compacting the CF that IS indexed (your CF), but the internal Cassandra's one, which is responsible for storing index data.
MBean you should look for looks like this:
W dniu 07.10.2013 15:22, Tom van den Berge pisze:
On a 2-node cluster with replication factor 2, I have a column family with
an index on one of the columns.
Every now and then, I notice that a lookup of the record through the index
on node 1 produces the record, but the same lookup on node 2 does not! If I
do a lookup by row key, the record is found, and the indexed value is there.
So as far as I can tell, the index on one of the nodes looses values, and
is no longer in sync with the other node, even though the replication
factor requires it. I typically repair these issues by storing the indexed
column value again.
The indexed data is static data; it doesn't change.
I'm running cassandra 1.2.3. I'm running a nodetool repair on each node
every day (although this does not fix this problem).
This problem worries me a lot. I don't have a clue about the cause of it.
Any help would be greatly appreciated.