couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Smith <...@iriscouch.com>
Subject Re: atomic action on replicated dbs?
Date Tue, 10 Jan 2012 01:46:32 GMT
I had similar requirements, and I decided to copy what Amazon does, in
particular, SQS, the Simple Queue System.  Amazon did the hard
thinking about the hard problems. In their design, they struck certain
trade-offs. I felt it was prudent to trust their judgement. In other
words, if it's good enough for Amazon AWS and amazon.com, it's good
enough for me.

That is why I implemented CQS, the CouchDB Queue System, which
provides an identical API as SQS.

    https://github.com/iriscouch/cqs

Notice that their messages have an "approximate receive count," not a
"for-sure receive count." One presumes this is to accommodate network
partitions, allowing conflicting changes (message receipts), and to
allow the receive count to reconstitute when you resolve the conflict.

Anyway, Iris Couch uses CQS in production and for our purposes it has
been wonderful. CQS uses CouchDB's MVCC design to check out and
complete jobs transactionally. Most of the time, it works perfectly.

If you write your task handlers to be idempotent and reentrant or
restartable, then the occasional fluke will not harm your operation.
(That's a good strategy regardless of how you implement your task
queue.)

On Tue, Jan 10, 2012 at 7:35 AM, Mark Hahn <mark@hahnca.com> wrote:
> I want to have multiple processes accessing multiple replicated servers.  I
> want a process to be able to pull a waiting task out of the db.  I don't
> want two processes to start work on the same task.
>
> Can someone suggest a design pattern to do this?  The algorithms I can
> think of are pretty messy and hard to prove are reliable.
>
> If I loosen the requirements so that two can start and then one is aborted,
> the problem is easier but I would like to avoid that.  Also I don't think
> it makes provability any easier.



-- 
Iris Couch

Mime
View raw message