The more I think about it, the more this feels like a column timestamp issue. If two inserts have the same timestamp then the values are compared lexically to decide which one to keep (which I think explains the "99"/"100" "999"/"1000" mystery). 

We can verify this by also selecting out the WRITETIME of the column:
 
...
var prevTS int
for i := 0; i < 10000; i++ {
val := fmt.Sprintf("%d", i)
db.Query("UPDATE ut.test SET val = ? WHERE key = 'foo'", val).Exec()

var result string
var ts int
db.Query("SELECT val, WRITETIME(val) FROM ut.test WHERE key = 'foo'").Scan(&result, &ts)
if result != val {
fmt.Printf("Expected %v but got: %v; (prevTS:%d, ts:%d)\n", val, result, prevTS, ts)
}
prevTS = ts
}


When I run it with this change I see that the timestamps are in fact the same:

Expected 10 but got: 9; (prevTS:1425345839903000, ts:1425345839903000)
Expected 100 but got: 99; (prevTS:1425345839939000, ts:1425345839939000)
Expected 101 but got: 99; (prevTS:1425345839939000, ts:1425345839939000)
Expected 1000 but got: 999; (prevTS:1425345840296000, ts:1425345840296000)


It looks like we're only getting millisecond precision instead of microsecond for the column timestamps?! If you explicitly set the timestamp value when you do the insert, you can get actual microsecond precision and the issue should go away.

-psanford

On Mon, Mar 2, 2015 at 4:21 PM, Dan Kinder <dkinder@turnitin.com> wrote:
Yeah I thought that was suspicious too, it's mysterious and fairly consistent. (By the way I had error checking but removed it for email brevity, but thanks for verifying :) )

On Mon, Mar 2, 2015 at 4:13 PM, Peter Sanford <psanford@retailnext.net> wrote:
Hmm. I was able to reproduce the behavior with your go program on my dev machine (C* 2.0.12). I was hoping it was going to just be an unchecked error from the .Exec() or .Scan(), but that is not the case for me.

The fact that the issue seems to happen on loop iteration 10, 100 and 1000 is pretty suspicious. I took a tcpdump to confirm that the gocql was in fact sending the "write 100" query and then on the next read Cassandra responded with "99".

I'll be interested to see what the result of the jira ticket is.

-psanford




--
Dan Kinder
Senior Software Engineer
Turnitin – www.turnitin.com
dkinder@turnitin.com