incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Gibson <kyle.gib...@frozenonline.com>
Subject Re: Occasionally getting old data back with ConsistencyLevel.ALL
Date Sun, 28 Aug 2011 23:11:46 GMT
Thanks for the reply Peter, you may have discovered the problem, I'll
explain below.

On Sun, Aug 28, 2011 at 8:51 AM, Peter Schuller
<peter.schuller@infidyne.com> wrote:
>> Understood. In the code example I provided, I am writing the same
>> value, but I am doing so in quick succession, so perhaps a few second
>> sleep might be helpful. It is worth noting also that the code I
>> provided is only the second step 2 in the process. There is a php
>> script that receives the post request from Paypal which inserts the
>> IPN data into the IPN column family. Before it does this, it sets the
>> "processed" column to "no"
>
> Is it at all possible that this step happens twice? I have no idea
> what Paypal does or document, but in general with an HTTP based
> callback you (in Paypal's position) would either have to accept that
> human intervention is necessary on any transaction where the callback
> fails, or else implement some kind of re-try and keep submitting to
> the customer until the callback is successful. Keep in mind that the
> other end (meaning you in this case) can perceive to receive a
> successful HTTP request and send back a response, even though Paypal
> may perceive an error on their end.

> If you haven't, I'd definitely recommend checking logs at this step,
> or adding logging if required, to make sure that the callback is not
> happening twice.

You are correct here. If PayPal fails to get a positive response from
my callback, it will retry the IPN event until it gets a successful
response. When this happens, a new column appears "retry_count" which
is set to the positive integer representing the number of tries
attempted. Given that this column has always shown as 0 and the IPN
event log on paypal.com also shows no retries attempted, I believe I
am correct in assuming that this isn't the case.

> How much traffic do you have to this cluster? Is it feasable to run
> Cassandra with full debug enabled (spammings lots of text in your
> logs)? That might be one way to ascertain, once you have one of these
> cases happening, whether Cassandra is mentioning any activity
> pertaining to the row that might explain this, such as it being
> re-written by a client.

Not really a lot of traffic, and if my fix below doesn't work I will
definitely give this a shot.

> Another suggestion: Is it possible you do not have clocks synchronized
> among your clients? Suppose that that Paypal *is* submitting twice
> sometimes, and e.g. one of your PHP front-ends (or whoever is talking
> to Cassandra to insert the data) has clock drift. This would render
> the insert from your code snippet obsolete, if there is already a
> value inserted with a timestamp in the future.

This appears to be the case. The server that the PHP front-end resides
on was 80 seconds into the future. The server that handles IPN
processing was sync'd with NTP to ntp.ubuntu.com. So if a processing
event occurred within 80 seconds of inserting the IPN event, and the
'processed' column was updated, the time stamp for that column would
be earlier than the original insert and thus obsolete. That's why it
always worked on the second attempt, because by then, enough time had
passed to make the original insert obsolete despite the drift.

For some reason the PHP front-end server was lacking any time
synchronization. I have corrected this, it now syncs to ntp.ubuntu.com
just like all the others. I will post back on this topic if it appears
to have solved the problem.

>
> --
> / Peter Schuller (@scode on twitter)
>

Mime
View raw message