Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAMZt1DdHv-ahzOUshodXAgHs_46RkndznB0oX5xETUPa5xzvCg@mail.gmail.com>
References: 
 <CAPb37+bAsRNWyd2xmmC9LAOBZ=wZOK_==Z7-Z0GhcQeeotVFGQ@mail.gmail.com>
 <CALdd-zgwyrjWTpnSvumR6kvGhoJ17YydvsQ1caz9VJnBY=7hSg@mail.gmail.com>
 <CAO5xsd1p_LpjHZkL2OORNAr+v-objaJHFLE-FB5D_E7-5P0vkg@mail.gmail.com>
 <CAPb37+YVUdchSL1jegZLjEN-V7fMYyBnR6jOMeu2WCDO+fZ-5Q@mail.gmail.com>
 <CAO5xsd0KobeLxPhFhE_gVtRUgJGJkQA6aCvYEJVzkcRZ_WKe3Q@mail.gmail.com>
 <CAPb37+ZBXcMzAoQ4tPkfzCjkbw2O6aSdoYFi+w7zjV6KXybuAg@mail.gmail.com>
 <CAMZt1DdHv-ahzOUshodXAgHs_46RkndznB0oX5xETUPa5xzvCg@mail.gmail.com>
From: Kyle Gibson <kyle.gibson@frozenonline.com>
Date: Mon, 22 Aug 2011 13:59:20 -0400
Message-ID: 
 <CAPb37+b6WTv0dYnsQHP48pycP_0C18BA8KTrHw8pi7BBZh3OSA@mail.gmail.com>
Subject: Re: Occasionally getting old data back with ConsistencyLevel.ALL
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the reply.

On Mon, Aug 22, 2011 at 1:11 PM, Dominic Williams
<dwilliams@fightmymonster.com> wrote:
> Hi there, here's my tuppence!
> 1. Something to look at first:
>
> If you write two different values to the same column quickly in succession,
> if both writes go out with the same timestamp, then it is indeterminate
> which one wins i.e. write order doesn't necessarily matter.

Understood. In the code example I provided, I am writing the same
value, but I am doing so in quick succession, so perhaps a few second
sleep might be helpful. It is worth noting also that the code I
provided is only the second step 2 in the process. There is a php
script that receives the post request from Paypal which inserts the
IPN data into the IPN column family. Before it does this, it sets the
"processed" column to "no"

> 2. Don't trust PayPal (anyone using PayPal should really read this)
> We are / were relying on IPNs to manage our website's recurring
> subscriptions list. We experienced this weird thing where the
> recurring_payment_profile_created IPN was missing, and got thought maybe
> Cassandra was losing it because PayPal is a financial system and it couldn't
> possibly fail to generate an IPN, right!!?
> Anyway, it turns out that after exhaustive discussions with PayPal
> engineers, and having proved this from the PayPal logs, that sometimes IPNs
> fail to get generated. Yup. Read that again!!!! Sometimes the fail to get
> generated and in fact this is happening to us quite regularly now.
> They justify this (while acknowledging this issue should be in their
> documentation) by saying that because HTTP delivery is unreliable (hmmm
> isn't this what the retry queue is for..) we shouldn't be relying entirely
> on IPNs and should regularly download the logs and run them through scripts
> to catch problems (this is idiotic, since the angry customer will get on our
> case immediately when they pay and membership doesn't start)
> Not sure whether PayPal or database failing is best option. Look forward to
> hearing resolution.

I have experienced a failing to receive an IPN event before. In this
case the IPN even is never saved to the IPN column family, and the
cron script doesn't process it once, or twice, for that matter. Odd
thing about the failed IPN event is that it didn't even show up in the
IPN history, so i couldn't "replay" the event.

I am fairly positive that the problem is either with my environment or
cassandra and not paypal in this case. I am hoping it is my
environment because i suspect that will be easier to fix.

Oddly enough, the second time the IPN is processed, the column write
succeeds. This always happens 5 minutes after the first one is
processed.

I neglected to mention an important part of the process: after the IPN
event is processed (e.g. a new payment), an email is sent out to
myself and the sender. This is how I know for sure the event is being
processed twice, because not only do I receive two emails (spaced 5
minutes apart) but does the individual who paid. This is often
embarrassing to explain and somewhat difficult, customers get confused
as to which account they are supposed to use, etc.

Thanks

> Best,
> Dominic
> On 22 August 2011 17:49, Kyle Gibson <kyle.gibson@frozenonline.com> wrote:
>>
>> I made some changes to my code base that uses cassandra. I went back
>> to using the "processed" column, but instead of using "0" or "1" I
>> decided to use "no" and "yes"
>>
>> You can view the code here: http://pastebin.com/gRBC16e7
>>
>> As you can see from the code, I perform an insert, get, check the
>> result, if it didn't work, I try to insert again, and check the get.
>> Each time I do a print out to see what the result is. Each operation
>> is a CL.ALL.
>>
>> A few successful IPNs did come through before this one was generated:
>>
>> IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce
>> OrderedDict([..., (u'processed', u'no'), ...])
>> Failed to set processed to yes
>> IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert 1314012603578714
>> Failed to set processed to yes
>> IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert2 1314012603586201
>>
>> As expected, this IPN was processed twice.
>>
>> On Sat, Aug 20, 2011 at 5:37 PM, Peter Schuller
>> <peter.schuller@infidyne.com> wrote:
>> >> Do you mean the cassandra log, or just logging in the script itself?
>> >
>> > The script itself. I.e, some "independent" verification that the line
>> > of code after the insert is in fact running, just in case there's some
>> > kind of silent failure.
>> >
>> > Sounds like you've tried to address it though with the E-Mail:s.
>> >
>> > I suppose it boils down to: Either there is something wrong in your
>> > environment/code, or Cassandra does have a bug. If the latter, it
>> > would probably be helpful if you could try to reproduce it in your
>> > environment in a way which can be shared - such as a script that does
>> > writes and reads back to confirm the write made it. Or maybe just
>> > adding more explicit logging to your script (even if it causes some
>> > log flooding) to "prove" that a write truly happened.
>> >
>> > --
>> > / Peter Schuller (@scode on twitter)
>> >
>
>