couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <>
Subject Re: Replacing the _external API
Date Fri, 24 Sep 2010 18:55:33 GMT
On Fri, Sep 24, 2010 at 2:48 PM, Zachary Zolton
<> wrote:
> Paul,
> Have you given any thought to having external processes being able to
> write to the CouchDB log, or show up in the _active_tasks?

For things managed by the process management, I was considering just
merging anything that came from stderr into the CouchDB logs. As to
_active_tasks, that's an interesting idea, if I implement the config
api of stdio, this could be an added fairly easily as well. Though for
things like lucene where Rob talks about having the lucene index on a
different machine, I don't think I'd allow for HTTP based access to
_active_tasks. Or rather, it'd be a lot harder because of how the
_active_tasks internals work.

> For example, it would be neat if I could watch the progress of Lucene
> indexing along side of building view indexes.
> That kind of integration makes it more compelling than just using
> other process management and proxy software.

The reason for this feature was always to simplify the configuration
overhead so that people can do interesting things without requiring a
full time sysadmin. I could do anything anyone's ever done with
externals using enough proxy fu and support code, it's just more

Though I do think the few extra features will be useful regardless.

Paul Davis

> Cheers,
> Zach
> On Fri, Sep 24, 2010 at 1:36 PM, Paul Davis <> wrote:
>> On Fri, Sep 24, 2010 at 2:28 PM, Stephen Prater <> wrote:
>>> As one of the people who wanted the external process management, that's a +1
>>> from me (if my vote counts.)
>> Every vote counts, its how we measure community consensus.
>>> But I like the sound of the reverse proxy protocol for externals too.
>>> stephen
>>> On Sep 24, 2010, at 1:19 PM, Robert Newson wrote:
>>>> Assuming it's straightforward to extend OTP-style process monitoring
>>>> to external processes (and I'm assuming that the couchjs processes are
>>>> so monitored today) then I like the proposal to add both of these
>>>> things.
>>>> My obvious motivation is couchdb-lucene so, with that hat on, would
>>>> this mechanism obviate the need for start couchdb-lucene externally
>>>> and make the Python hook script obsolete? I think it does. Finally,
>>>> there are cases where c-l users might wish to locate their c-l server
>>>> on a different box, so we should allow the proxying independently of
>>>> the launch-on-demand-and-keep-me-running bit.
>>>> B.
>>>> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <>
>>>> wrote:
>>>>> At CouchCamp there was a bit of discussion on replacing the _external
>>>>> API with something a bit more modern to give _external processes more
>>>>> control over their environment.
>>>>> The idea was born out of a discussion with Robert Newson who mentioned
>>>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>>>> the same URL namespace. It occurred to us that having a reverse proxy
>>>>> instead of the current _external stdio protocol would allow lots of
>>>>> other interesting features like node.js integration, as well as allow
>>>>> implementors to handle requests in parallel and so on and such forth.
>>>>> The major drawback that was identified was that if we switched to just
>>>>> a reverse proxy, people would then be responsible for handling the
>>>>> process management of their _external handlers. Ie, they'd have to
>>>>> configure daemon monitoring to make sure the processes stayed up and
>>>>> what not. The solution we came up with was to include another feature
>>>>> that did process management. Ie, something that would bring up an OS
>>>>> process when the server booted, and respawn it if it crashed. There'd
>>>>> be no connection to the _externals. Other than the basic "just keep a
>>>>> process up" sort of behaviour, the only other thing I could see adding
>>>>> is a simple stdio protocol to get configuration values from CouchDB.
>>>>> Other people have expressed interest in just the process management
>>>>> functionality as well which makes me think that having the two new
>>>>> features to replace the _external API would be both easier on
>>>>> developers as well as providing more functionality.
>>>>> So now I'm looking for feedback on what other people might think of
>>>>> this. I'll start working on this fairly soon if I don't hear any major
>>>>> objections.
>>>>> HTH,
>>>>> Paul Davis

View raw message