Background: My application is running on a cluster system(which have 4 nodes), and system time of these four nodes are synchronizing by NTP. I use Write.QUORUM and Read.QUORUM strategy. The probability of this problem is not very high. Cassandra version is 1.0.3, I have tried Cassandra 1.1.1, this problem is still exist.

Problem: I deleted a column, but after 6 seconds, Cassandra can still get the old record which "isMarkedForDelete" is still false.

Is anybody meet the same problem? And how to solve it?

Detail: See the log below:

Node 3(Local node):
[pool-2-thread-42] 2012-06-27 14:49:23,732 SliceQueryFilter.java (line 123) collecting 0 of 2147483647: SuperColumn(667072 [......7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,......)

[pool-2-thread-44] 2012-06-27 14:51:21,367 StorageProxy.java (line 172) Mutations/ConsistencyLevel are [RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])])]/QUORUM

-- I delete this record at 14:51:21,367

[pool-2-thread-37] 2012-06-27 14:51:27,400 SliceQueryFilter.java (line 123) collecting 0 of 2147483647: SuperColumn(667072 [......,7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,......)

-- But I can still get the old record at 14:51:27,400

Node2:
[MutationStage:118] 2012-06-27 14:51:21,373 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])])

[MutationStage:118] 2012-06-27 14:51:21,374 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])]) applied. Sending response to 6692098@/192.168.0.3

[MutationStage:123] 2012-06-27 14:51:27,405 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [......,7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,......])

[MutationStage:123] 2012-06-27 14:51:27,405 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [......,7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,.......]),])]) applied. Sending response to 6698516@/192.168.0.3

Node1:
[MutationStage:98] 2012-06-27 14:51:24,661 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])])

[MutationStage:98] 2012-06-27 14:51:24,675 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698atrue :4@1340779881338000,]),])]) applied. Sending response to 6692099@/192.168.0.3

[MutationStage:93] 2012-06-27 14:51:40,932 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779900915004,]),])])

DEBUG [MutationStage:93] 2012-06-27 14:51:40,933 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698atrue :4@1340779900915004,]),])]) applied. Sending response to 6706555@/192.168.0.3

[ReadStage:55] 2012-06-27 14:51:43,074 SliceQueryFilter.java (line 123) collecting 0 of 5000:7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779900915004

Node 4:

There is no log about this record on Node 4.