couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Anderson <>
Subject Re: Document validation involving other documents
Date Sun, 24 Jan 2010 23:37:57 GMT
On Sun, Jan 24, 2010 at 2:25 PM, David Richardson <> wrote:
> That's exactly what creates the high associated procedure cost - the documents are submitted
by mobile workers who connect for very short times, sometimes very infrequently. If the document
is accepted with invalid entries and processed later, the opportunity to alert the author
of the problem(s) during the request/response cycle is lost. This means someone else must
fix the mistake (extra work), and a very unwieldy process (usually manual) must be instituted
to alert the author (training/feedback is harder and less effective).
> I'm now willing to accept extra moving parts living in front of the db - there are other
things that are easier to accomplish there too.
> The cron async process smells like Greenspun's 10th law, extrapolated to transaction
managers and message brokers ;-)

The advantage of having an asynchronous document state machine handler
is a whole lot of decoupling, as the queue is just a side effect of
the database.

>>> I've been trying to eliminate, or at least vastly reduce the need for middleware
in a specific class of applications, but this problem probably cuts that task short.

I'm wary of the procedural-controller middleware approach because I'm
afraid it encourages you to program as though you have transactions.
(Eg: change these 4 documents at the same time and hope for the best.)

If you are careful with asynchronous handlers, you can be careful to
pass the state through the database _changes feeds in a way that
doesn't give devs a false sense of multi-document transactions.

This is why I think an asynchronous node.js _changes handler (or
browser ajax handlers) or event based programming in any language is
such a good fit for CouchDB. If you transmit all of your important
events through the database, even real-time apps can scale by using

Each doc update is atomic, so there's room to model larger
transactions as a series of atomic updates.


>> On Sun, Jan 24, 2010 at 12:06 PM, David Richardson
>> <> wrote:
>>> I've been trying to eliminate, or at least vastly reduce the need for middleware
in a specific class of applications, but this problem probably cuts that task short.
>>> We have incoming documents that need to be validated against data stored in other
docs (rate tables, technical abbreviations, inspection codes). There is a high procedural
cost to allowing documents in and validating them after saving (queue processing), but validation
functions don't allow bringing in external data (correct me if I'm wrong, please.) Without
writing middleware or sideware (an external handler), I don't see a way to do this.
>>> What are the arguments against including a native 'lookup' function available
to validation functions only. This seems to be a side-effect-free usage case to me. Reads
are cheap, correct?
>>> If my memory hasn't completely failed, this was a significant problem in Lotus
Notes until such a function was added, somewhere around version 3.3 or 4.0.
>>> David Richardson
>> The problem with this approach is that validation is run during
>> replication as well, so any multi-doc data dependencies become
>> problematic in ad-hoc clusters.
>> The way to do this without middleware is to have a backend
>> asynchronous process (maybe node.js) that consumes _changes and acts
>> on particular updates. So a user saves the doc with a pending state
>> and the _changes handler sees that and initiates validation.
>> (One trick you can do with this is use _changes heartbeat to generate
>> a cron trigger, so your async process can also do things based on
>> time.)
>> I'd like to see this async/cron functionality bundled with Couch
>> itself, but there's only so much time in the day and I don't think
>> it's worth delaying any releases for. Patches extremely welcome, but I
>> think this one might require more discussion before anyone would be
>> ready to run off and write it.
>> Chris
>> --
>> Chris Anderson

Chris Anderson

View raw message