incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Gibson <>
Subject Occasionally getting old data back with ConsistencyLevel.ALL
Date Thu, 18 Aug 2011 14:49:18 GMT
I am running cassandra 0.7.8. pycassa 1.1.0

Nodes=7, RF=3

This problem started a few months ago and only occurs sporadically.

I receive notifications from paypal's IPN. The IPN data is saved into
a column family. I add another column for "processed" which is set to

Every 5 minutes, a cron script runs and pulls down IPN events that
haven't been processed. It does some work, and then writes back
processed to 1.

Usually this worked, but then (when exactly I don't recall),
occasionally, it started having a problem. Processed wasn't been set
to 1. So, IPN events would be processed twice.

I tried a few things to fix this. Repair, compact, restarting the
cluster, upgrading. I even did a complete rebuild of the cluster,
wiping the data directory and starting fresh on 0.7.8.

I then ditched the "processed" column and decide to use two column
families. IPNs are put into column family A, and after being
processed, it is inserted into column family B and deleted from A.

The problem still persisted. At this point I was using CL.QUORUM. So,
I started using CL.ALL.

And the problem still persists. Having IPN events processed twice
causes a fair bit of problems, so this is something I really need to
get resolved.



View raw message