cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Sanford <>
Subject Re: Reboot: Read After Write Inconsistent Even On A One Node Cluster
Date Tue, 03 Mar 2015 01:34:04 GMT
The more I think about it, the more this feels like a column timestamp
issue. If two inserts have the same timestamp then the values are compared
lexically to decide which one to keep (which I think explains the
"99"/"100" "999"/"1000" mystery).

We can verify this by also selecting out the WRITETIME of the column:

var prevTS int
for i := 0; i < 10000; i++ {
val := fmt.Sprintf("%d", i)
db.Query("UPDATE ut.test SET val = ? WHERE key = 'foo'", val).Exec()

var result string
var ts int
db.Query("SELECT val, WRITETIME(val) FROM ut.test WHERE key =
'foo'").Scan(&result, &ts)
if result != val {
fmt.Printf("Expected %v but got: %v; (prevTS:%d, ts:%d)\n", val, result,
prevTS, ts)
prevTS = ts

When I run it with this change I see that the timestamps are in fact the

Expected 10 but got: 9; (prevTS:1425345839903000, ts:1425345839903000)
Expected 100 but got: 99; (prevTS:1425345839939000, ts:1425345839939000)
Expected 101 but got: 99; (prevTS:1425345839939000, ts:1425345839939000)
Expected 1000 but got: 999; (prevTS:1425345840296000, ts:1425345840296000)

It looks like we're only getting millisecond precision instead of
microsecond for the column timestamps?! If you explicitly set the timestamp
value when you do the insert, you can get actual microsecond precision and
the issue should go away.


On Mon, Mar 2, 2015 at 4:21 PM, Dan Kinder <> wrote:

> Yeah I thought that was suspicious too, it's mysterious and fairly
> consistent. (By the way I had error checking but removed it for email
> brevity, but thanks for verifying :) )
> On Mon, Mar 2, 2015 at 4:13 PM, Peter Sanford <>
> wrote:
>> Hmm. I was able to reproduce the behavior with your go program on my dev
>> machine (C* 2.0.12). I was hoping it was going to just be an unchecked
>> error from the .Exec() or .Scan(), but that is not the case for me.
>> The fact that the issue seems to happen on loop iteration 10, 100 and
>> 1000 is pretty suspicious. I took a tcpdump to confirm that the gocql was
>> in fact sending the "write 100" query and then on the next read Cassandra
>> responded with "99".
>> I'll be interested to see what the result of the jira ticket is.
>> -psanford
> --
> Dan Kinder
> Senior Software Engineer
> Turnitin –

View raw message