couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nicholas a. evans" <>
Subject Re: Options for Iterative Map Reduce
Date Wed, 12 Dec 2012 23:07:31 GMT
On Wed, Dec 12, 2012 at 4:23 PM, Alon Keren <> wrote:
> couch-incarnate author here.
> You described the way it works pretty well.

Thanks Alon, I did a lot of source inspection and poking and prodding
to make sure I understood what was going on.  :)  couch-incarnate was
very helpful for me to understand what needs to be done in order to do
this right.

> I'm a bit biased here, but I find the end solution to be pretty relaxed and
> generally with the grain of couch: the service is HTTP-based,
> eventually-consistent and persistent (its state is kept only on CouchDB).

I think you are more or less correct.  I was evaluating it more on the
relaxed nature of the algorithm and architecture than on the
end-user's API.  I was also convinced there had to be a simpler (two
databases) implementation, but since I sent my first email I've become
convinced that your basic approach (three databases) is the only way
to get an incremental iterative map reduce from CouchDB (without
poking around in the Erlang innards).

> Unfortunately, my work took me away from couch, and aside from a single
> issue [1] there was little interest in the project from others.
> It'd be nice to hear if dbcopy or any other chaining solution is going to
> finally become a part of couch.
> If not, and if you or anyone else wish to see some effort put into
> couch-incarnate (see todo's and caveats in the readme), feel free to
> contact me.

Unfortunately, those TODOs and Caveats are part of what nix
couch-incarnate for us. "input validation"; I made it crash several
times (with invalid JSON) while I was testing it out. "limited
error-handling" and "tests. Does this thing really work?";
self-explanatory.  ;)  The crashing plus the lack of recent commits
made me think it was more of a (damn fine) proof of concept or a beta
release than a production hardened code base.

I had a hard enough time reading through the node.js twisty-turny
evented code that I figured it would be hard for *me* to support it in
production.  I've spent months doing evented programming with ruby's
EventMachine, and eventually came up with a few techniques to keep
myself sane.  And there are the minor deployment annoyances of adding
node.js to every server and making sure that yet another daemon is
running at all times.  None of these are individually deal breakers.
But all of them together made me want to see if I could write
something myself that I could support in production, using your
architecture as inspiration.

At any rate, I need an incremental iterative map reduce.  And although
we may use Cloudant at some point in the future, locking ourselves to
them isn't an option for us right now.  So I will continue to poke and
prod at couch-incarnate this week; either as a reference while I build
a library inspired by it, or as a foundation while I start using it.


View raw message