couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Samuel Newson <rnew...@apache.org>
Subject Re: npm and couchdb
Date Wed, 12 Feb 2014 11:43:06 GMT

I can’t think of a way to say that couchdb is still used but behind a layer to work around
perceived deficiencies, so I would personally say nothing.

The story is deeper and wider than I know but I do know that at least some of the issues encountered
here are to do with the approach taken, rather than couchdb being entirely culpable. Mikeal
posted recently about the history of the architecture here and seems to confirm that. Had
something like the global changes feed existed at the time and a database-per-package approach
been taken, I think the story would have been radically different.

I would rather focus on the actual problems in couchdb rather than the problems in the original
npm architectures use of npm; that plane has sailed.

The main problems are around the replicator. It’s a complicated piece of code (and without
technical justification for that complexity, it’s our failure to simplify) and can fail
to make progress under some conditions (notably, and pertinently, attachment heavy databases).
The replicator, for all its complexity, does not make some obvious optimizations. A partially
transferred attachment followed by a connection break causes the full re-transfer of the attachment.
This can be a real killer and I believe was a real issue in npm registry replication. If a
document is updated, all the attachments are replicated again, even if they exist on the target.

CouchDB *ought* to have been the perfect solution to the problem of mirroring a repository
such as npm. We have made significant improvements to the replicator in the last several releases,
all of which made npm replication better. That work should continue, and I think it’s time
for a new replicator implementation; we can do better.

A further improvement, that Volker has certainly attempted recently with some (but not total)
success, would be to allow replication to work from a static host server. Currently, replication
requires couchdb, or couchdb-like system, at source and target. That enables some important
optimizations but if it were not mandatory it would open up some interesting possibilities.

B.

On 12 Feb 2014, at 11:12, Benoit Chesneau <bchesneau@gmail.com> wrote:

> Some people may have noticed it but npm inc is now live for real and
> has raised $ 2.5 M [1]. It may be a good time to communicate that
> couchdb even behind a layer is in used in npm. Also maybe Jason has
> some interesting stories to give. Maybe one that know how to write can
> take it ;)
> 
> I also found that this  other post  in their blog [2] can be
> interesting for other couchdb users and dev.  Not saying that novacut
> is taking a similar approach though different (not using any private
> layer) [3] in demedia.
> 
> 
> Just saying.
> 
> - benoit
> 
> [1] http://blog.npmjs.org/post/76320673650/funding
> [2] http://blog.npmjs.org/post/75707294465/new-npm-registry-architecture
> [3] https://launchpad.net/dmedia


Mime
View raw message