cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject Consistency level and ReadRepair
Date Wed, 05 Oct 2011 17:37:04 GMT
I have a 12 node cassandra cluster running with RF=3.  I have severl
clients ( all running on a single node ) connecting to the cluster (
fixed client - node mapping ) and try to do a insert, update , select
and delete. Each client has a fixed mapping of the row-keys and always
connect to the same node. The timestamp on the client node is used for
all operations.  All operations are done using CL QUORUM.

When  I run a tpstats I see the ReadRepair count consistently
increasing. i need to figure out why ReadRepair is happening..

One scenario I can think of is, it could happen when there is a delay
in updating the nodes to reach eventual consistency..

Let's say I have 3 nodes (RF=3)  A,B,C. I insert  <key> with timestamp
<ts1> to A and the call will return as soon as it inserts the record
to A and B. At some later point this information is sent to C...

A while later A,B,C have the same data with the same timestamp.

A <key,ts1>
B <key, ts1> and
C <key, ts1>

When I update <key> on A with timestamp <ts2> to A, the call will
return as soon as it inserts the record to A and B.
Now the data is

A <key,ts2>
B <key,ts2>
C <key,ts1>

Assuming I query for <key>  A,C respond and since there is no QUORUM,
it waits for B to respond and when A,B match, the response is returned
to the client and ReadRepair is sent to C.

This could happen only when C is running behind in catching up the
updates to A,B.  Are there any stats that would let me know if the
system is in a consistent state?

thanks
Ramesh


tpstats_2011-10-05_12:50:01:ReadRepairStage                   0
 0       43569781         0                 0
tpstats_2011-10-05_12:55:01:ReadRepairStage                   0
 0       43646420         0                 0
tpstats_2011-10-05_13:00:02:ReadRepairStage                   0
 0       43725850         0                 0
tpstats_2011-10-05_13:05:01:ReadRepairStage                   0
 0       43790047         0                 0
tpstats_2011-10-05_13:10:02:ReadRepairStage                   0
 0       43869704         0                 0
tpstats_2011-10-05_13:15:01:ReadRepairStage                   0
 0       43945635         0                 0
tpstats_2011-10-05_13:20:01:ReadRepairStage                   0
 0       44020406         0                 0
tpstats_2011-10-05_13:25:02:ReadRepairStage                   0
 0       44093227         0                 0
tpstats_2011-10-05_13:30:01:ReadRepairStage                   0
 0       44167455         0                 0
tpstats_2011-10-05_13:35:02:ReadRepairStage                   0
 0       44247519         0                 0
tpstats_2011-10-05_13:40:01:ReadRepairStage                   0
 0       44312726         0                 0
tpstats_2011-10-05_13:45:01:ReadRepairStage                   0
 0       44387633         0                 0
tpstats_2011-10-05_13:50:01:ReadRepairStage                   0
 0       44443683         0                 0
tpstats_2011-10-05_13:55:02:ReadRepairStage                   0
 0       44499487         0                 0
tpstats_2011-10-05_14:00:01:ReadRepairStage                   0
 0       44578656         0                 0
tpstats_2011-10-05_14:05:01:ReadRepairStage                   0
 0       44647555         0                 0
tpstats_2011-10-05_14:10:02:ReadRepairStage                   0
 0       44716730         0                 0
tpstats_2011-10-05_14:15:01:ReadRepairStage                   0
 0       44776644         0                 0
tpstats_2011-10-05_14:20:01:ReadRepairStage                   0
 0       44840237         0                 0
tpstats_2011-10-05_14:25:01:ReadRepairStage                   0
 0       44891444         0                 0
tpstats_2011-10-05_14:30:01:ReadRepairStage                   0
 0       44931105         0                 0
tpstats_2011-10-05_14:35:02:ReadRepairStage                   0
 0       44976801         0                 0
tpstats_2011-10-05_14:40:01:ReadRepairStage                   0
 0       45042220         0                 0
tpstats_2011-10-05_14:45:01:ReadRepairStage                   0
 0       45112141         0                 0
tpstats_2011-10-05_14:50:02:ReadRepairStage                   0
 0       45177816         0                 0
tpstats_2011-10-05_14:55:02:ReadRepairStage                   0
 0       45246675         0                 0
tpstats_2011-10-05_15:00:01:ReadRepairStage                   0
 0       45309533         0                 0
tpstats_2011-10-05_15:05:01:ReadRepairStage                   0
 0       45357575         0                 0
tpstats_2011-10-05_15:10:01:ReadRepairStage                   0
 0       45405943         0                 0
tpstats_2011-10-05_15:15:01:ReadRepairStage                   0
 0       45458435         0                 0
tpstats_2011-10-05_15:20:01:ReadRepairStage                   0
 2       45508253         0                 0
tpstats_2011-10-05_15:25:01:ReadRepairStage                   0
 0       45570375         0                 0
tpstats_2011-10-05_15:30:01:ReadRepairStage                   0
 0       45628426         0                 0
tpstats_2011-10-05_15:35:01:ReadRepairStage                   0
 0       45688694         0                 0
tpstats_2011-10-05_15:40:01:ReadRepairStage                   0
 3       45743029         0                 0
tpstats_2011-10-05_15:45:02:ReadRepairStage                   0
 0       45801167         0                 0
tpstats_2011-10-05_15:50:02:ReadRepairStage                   0
 0       45837329         0                 0
tpstats_2011-10-05_15:55:01:ReadRepairStage                   0
 0       45890326         0                 0
tpstats_2011-10-05_16:00:01:ReadRepairStage                   0
 0       45951703         0                 0
tpstats_2011-10-05_16:05:02:ReadRepairStage                   0
 0       46010736         0                 0
tpstats_2011-10-05_16:10:01:ReadRepairStage                   0
 0       46063294         0                 0
tpstats_2011-10-05_16:15:01:ReadRepairStage                   0
 0       46108327         0                 0
tpstats_2011-10-05_16:20:01:ReadRepairStage                   0
 0       46142291         0                 0

Mime
View raw message