cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Kinder <dkin...@turnitin.com>
Subject Re: Reboot: Read After Write Inconsistent Even On A One Node Cluster
Date Tue, 03 Mar 2015 18:08:12 GMT
Per Aleksey Yeschenko's comment on that ticket, it does seem like a
timestamp granularity issue, but it should work properly if it is within
the same session. gocql by default uses 2 connections and 128 streams per
connection. If you set it to 1 connection with 1 stream this problem goes
away. I suppose that'll take care of it in testing.

At least one interesting conclusion here: a gocql.Session does not map to
one Cassandra "session". This makes some sense given that gocql says to use
Session shared concurrently (so it better not just be one Cassandra
session), but it is a bit concerning that there is no way to make this 100%
safe outside of cutting the gocql.Session down to 1 connection and stream.

On Mon, Mar 2, 2015 at 5:34 PM, Peter Sanford <psanford@retailnext.net>
wrote:

> The more I think about it, the more this feels like a column timestamp
> issue. If two inserts have the same timestamp then the values are compared
> lexically to decide which one to keep (which I think explains the
> "99"/"100" "999"/"1000" mystery).
>
> We can verify this by also selecting out the WRITETIME of the column:
>
> ...
> var prevTS int
> for i := 0; i < 10000; i++ {
> val := fmt.Sprintf("%d", i)
> db.Query("UPDATE ut.test SET val = ? WHERE key = 'foo'", val).Exec()
>
> var result string
> var ts int
> db.Query("SELECT val, WRITETIME(val) FROM ut.test WHERE key =
> 'foo'").Scan(&result, &ts)
> if result != val {
> fmt.Printf("Expected %v but got: %v; (prevTS:%d, ts:%d)\n", val, result,
> prevTS, ts)
> }
> prevTS = ts
> }
>
>
> When I run it with this change I see that the timestamps are in fact the
> same:
>
> Expected 10 but got: 9; (prevTS:1425345839903000, ts:1425345839903000)
> Expected 100 but got: 99; (prevTS:1425345839939000, ts:1425345839939000)
> Expected 101 but got: 99; (prevTS:1425345839939000, ts:1425345839939000)
> Expected 1000 but got: 999; (prevTS:1425345840296000, ts:1425345840296000)
>
>
> It looks like we're only getting millisecond precision instead of
> microsecond for the column timestamps?! If you explicitly set the timestamp
> value when you do the insert, you can get actual microsecond precision and
> the issue should go away.
>
> -psanford
>
> On Mon, Mar 2, 2015 at 4:21 PM, Dan Kinder <dkinder@turnitin.com> wrote:
>
>> Yeah I thought that was suspicious too, it's mysterious and fairly
>> consistent. (By the way I had error checking but removed it for email
>> brevity, but thanks for verifying :) )
>>
>> On Mon, Mar 2, 2015 at 4:13 PM, Peter Sanford <psanford@retailnext.net>
>> wrote:
>>
>>> Hmm. I was able to reproduce the behavior with your go program on my dev
>>> machine (C* 2.0.12). I was hoping it was going to just be an unchecked
>>> error from the .Exec() or .Scan(), but that is not the case for me.
>>>
>>> The fact that the issue seems to happen on loop iteration 10, 100 and
>>> 1000 is pretty suspicious. I took a tcpdump to confirm that the gocql was
>>> in fact sending the "write 100" query and then on the next read Cassandra
>>> responded with "99".
>>>
>>> I'll be interested to see what the result of the jira ticket is.
>>>
>>> -psanford
>>>
>>>
>>
>>
>> --
>> Dan Kinder
>> Senior Software Engineer
>> Turnitin – www.turnitin.com
>> dkinder@turnitin.com
>>
>
>


-- 
Dan Kinder
Senior Software Engineer
Turnitin – www.turnitin.com
dkinder@turnitin.com

Mime
View raw message