couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Anderson <jch...@apache.org>
Subject Re: ddoc protocol for query server
Date Mon, 21 Dec 2009 05:35:20 GMT
Devs,

My design doc query server branch is now ready for commit. It
maintains backwards compatibility, with some small improvements to
query handling, across all the APIs except for the query server
protocol. It does not effect the map-reduce part of the query server
protocol.

The current branch can be found here:

http://github.com/jchris/couchdb/tree/ddoc-qs

I'm also attaching an (almost current) snapshot of the diff to:

https://issues.apache.org/jira/browse/COUCHDB-589

I've got all the tests passing, and moved the native query servers to
a gen_server (although I didn't completely eliminate use of the
process dictionary, this will not be hard now). There are other
goodies in here, like better query server error handling. I've also
refactored the JavaScript code to avoid global variables, although we
could do more work here on the sandboxing.

I intend to commit this patch soon. It's a lot of lines, so I want to
get it in the next couple of days. I don't think it breaks anything,
but it's still better to let it sit in trunk for a while before the
release.

The only people this change should effect are those using alternate
query servers for more than just map-reduce. If you are one of these
people, please inspect the branch and let me know how you feel about
it.

The gen_server around the native query server could use some review,
too. It all seems to work just fine but there's nothing really in
there about when the server should be shut down. I think that's fine
as long as it goes down on config change, which it seems to.

The way design docs are cached inside the server simplifies function
dispatch and fixes the one os-process per filtered _changes feed
problem. It would be good to profile list / show / update / validation
now. They should be faster as they involve less data sent between
Couch and the query server.

One concern is that on systems with a large number of distinct design
documents (or very large design documents) the memory utilization
could go up. Currently attachments are not loaded into the design
document for this reason.

Only the latest version of each design document is stored in the query
server, so if you have 2 dbs with the same ddoc at different revs,
it's possible that some flapping could occur. The alternative is
storing ddocs by rev and using a JavaScript LRU cache. That'd be a
worthy improvement but it can come later.

The patch opens the door for more flexibility within the JavaScript
query server. For instance, CouchDB will be able to support APIs like
the require() statement from CommonJS:

http://commonjs.org/specs/modules/1.0.html

Chris

On Thu, Dec 17, 2009 at 6:39 PM, Chris Anderson <jchris@apache.org> wrote:
> I've been working on a medium-sized branch for a while and I'm finally
> getting something worth showing... almost.
>
> I'm developing on http://github.com/jchris/couchdb/tree/ddoc-qs
>
> The essential difference is in the way the code is handled by the
> query server. The query server keeps a cache of design documents
> around. This allows us to simplify the programming model for show,
> list, update, etc, and also I think will make it easier to add new
> commands in the future.
>
> My aim was to cut a lot of code, but stay close to a pure refactoring.
> So I haven't substantially changed the Futon test suite. But I did
> make some changes to query_server_spec.rb.
>
> If anyone would like to pull the Erlang query server into line with
> these changes, I'd appreciate the help.
>
> The branch is still deeply messy in some places. When it comes time to
> push to CouchDB trunk, I'll hand-edit the patch for log levels and
> inane comments.
>
> Chris
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Mime
View raw message