Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8B10F8D87 for ; Mon, 22 Aug 2011 17:12:28 +0000 (UTC) Received: (qmail 25604 invoked by uid 500); 22 Aug 2011 17:12:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 25290 invoked by uid 500); 22 Aug 2011 17:12:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 25278 invoked by uid 99); 22 Aug 2011 17:12:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Aug 2011 17:12:24 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dwilliams@system7.co.uk designates 209.85.210.174 as permitted sender) Received: from [209.85.210.174] (HELO mail-iy0-f174.google.com) (209.85.210.174) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Aug 2011 17:12:19 +0000 Received: by iyf40 with SMTP id 40so11645952iyf.33 for ; Mon, 22 Aug 2011 10:11:59 -0700 (PDT) Received: by 10.231.41.147 with SMTP id o19mr5901501ibe.82.1314033119071; Mon, 22 Aug 2011 10:11:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.47.129 with HTTP; Mon, 22 Aug 2011 10:11:38 -0700 (PDT) In-Reply-To: References: From: Dominic Williams Date: Mon, 22 Aug 2011 18:11:38 +0100 Message-ID: Subject: Re: Occasionally getting old data back with ConsistencyLevel.ALL To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001517741b9e490e1904ab1b2de1 --001517741b9e490e1904ab1b2de1 Content-Type: text/plain; charset=ISO-8859-1 Hi there, here's my tuppence! 1. Something to look at first: If you write two different values to the same column quickly in succession, if both writes go out with the same timestamp, then it is indeterminate which one wins i.e. write order doesn't necessarily matter. 2. Don't trust PayPal (anyone using PayPal should really read this) We are / were relying on IPNs to manage our website's recurring subscriptions list. We experienced this weird thing where the recurring_payment_profile_created IPN was missing, and got thought maybe Cassandra was losing it because PayPal is a financial system and it couldn't possibly fail to generate an IPN, right!!? Anyway, it turns out that after exhaustive discussions with PayPal engineers, and having proved this from the PayPal logs, that sometimes IPNs fail to get generated. Yup. Read that again!!!! Sometimes the fail to get generated and in fact this is happening to us quite regularly now. They justify this (while acknowledging this issue should be in their documentation) by saying that because HTTP delivery is unreliable (hmmm isn't this what the retry queue is for..) we shouldn't be relying entirely on IPNs and should regularly download the logs and run them through scripts to catch problems (this is idiotic, since the angry customer will get on our case immediately when they pay and membership doesn't start) Not sure whether PayPal or database failing is best option. Look forward to hearing resolution. Best, Dominic On 22 August 2011 17:49, Kyle Gibson wrote: > I made some changes to my code base that uses cassandra. I went back > to using the "processed" column, but instead of using "0" or "1" I > decided to use "no" and "yes" > > You can view the code here: http://pastebin.com/gRBC16e7 > > As you can see from the code, I perform an insert, get, check the > result, if it didn't work, I try to insert again, and check the get. > Each time I do a print out to see what the result is. Each operation > is a CL.ALL. > > A few successful IPNs did come through before this one was generated: > > IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce > OrderedDict([..., (u'processed', u'no'), ...]) > Failed to set processed to yes > IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert 1314012603578714 > Failed to set processed to yes > IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert2 1314012603586201 > > As expected, this IPN was processed twice. > > On Sat, Aug 20, 2011 at 5:37 PM, Peter Schuller > wrote: > >> Do you mean the cassandra log, or just logging in the script itself? > > > > The script itself. I.e, some "independent" verification that the line > > of code after the insert is in fact running, just in case there's some > > kind of silent failure. > > > > Sounds like you've tried to address it though with the E-Mail:s. > > > > I suppose it boils down to: Either there is something wrong in your > > environment/code, or Cassandra does have a bug. If the latter, it > > would probably be helpful if you could try to reproduce it in your > > environment in a way which can be shared - such as a script that does > > writes and reads back to confirm the write made it. Or maybe just > > adding more explicit logging to your script (even if it causes some > > log flooding) to "prove" that a write truly happened. > > > > -- > > / Peter Schuller (@scode on twitter) > > > --001517741b9e490e1904ab1b2de1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi there, here's my tuppence!

1. Something to look a= t first:

If you write two different values to the sa= me column quickly in succession, if both writes go out with the same timest= amp, then it is indeterminate which one wins i.e. write order doesn't n= ecessarily matter.

2. Don't trust PayPal (anyone using PayPal should r= eally read this)

We are / were relying on IPNs to = manage our website's recurring subscriptions list. We experienced this = weird thing where the recurring_payment_profile_created IPN was missing, an= d got thought maybe Cassandra was losing it because PayPal is a financial s= ystem and it couldn't possibly fail to generate an IPN, right!!?

Anyway, it turns out that after exhaustive discussions = with PayPal engineers, and having proved this from the PayPal logs, that so= metimes IPNs fail to get generated. Yup. Read that again!!!! Sometimes the = fail to get generated and in fact this is happening to us quite regularly n= ow.

They justify this (while acknowledging this issue shoul= d be in their documentation) by saying that because HTTP delivery is unreli= able (hmmm isn't this what the retry queue is for..) we shouldn't b= e relying entirely on IPNs and should regularly download the logs and run t= hem through scripts to catch problems (this is idiotic, since the angry cus= tomer will get on our case immediately when they pay and membership doesn&#= 39;t start)

Not sure whether PayPal or database failing is best opt= ion. Look forward to hearing resolution.

Best,=A0<= /div>
Dominic

On 22 Augus= t 2011 17:49, Kyle Gibson <kyle.gibson@frozenonline.com> wrote:
I made some changes to my code base that us= es cassandra. I went back
to using the "processed" column, but instead of using "0&quo= t; or "1" I
decided to use "no" and "yes"

You can view the code here: http://pastebin.com/gRBC16e7

As you can see from the code, I perform an insert, get, check the
result, if it didn't work, I try to insert again, and check the get. Each time I do a print out to see what the result is. Each operation
is a CL.ALL.

A few successful IPNs did come through before this one was generated:

IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce
OrderedDict([..., (u'processed', u'no'), ...])
Failed to set processed to yes
IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert 1314012603578714
Failed to set processed to yes
IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert2 1314012603586201

As expected, this IPN was processed twice.

On Sat, Aug 20, 2011 at 5:37 PM, Peter Schuller
<peter.schuller@infidyne.= com> wrote:
>> Do you mean the cassandra log, or just logging in the script itsel= f?
>
> The script itself. I.e, some "independent" verification that= the line
> of code after the insert is in fact running, just in case there's = some
> kind of silent failure.
>
> Sounds like you've tried to address it though with the E-Mail:s. >
> I suppose it boils down to: Either there is something wrong in your > environment/code, or Cassandra does have a bug. If the latter, it
> would probably be helpful if you could try to reproduce it in your
> environment in a way which can be shared - such as a script that does<= br> > writes and reads back to confirm the write made it. Or maybe just
> adding more explicit logging to your script (even if it causes some > log flooding) to "prove" that a write truly happened.
>
> --
> / Peter Schuller (@scode on twitter)
>

--001517741b9e490e1904ab1b2de1--