couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <>
Subject Re: NPM, CouchDB and big attachments
Date Wed, 27 Nov 2013 14:14:04 GMT
"replicate 100GB (and growing) data just to have a local npm web service."

This makes no sense to me. Of course you have to have a local copy of
all the things to have a "local npm web service". Are we talking about
partial mirroring instead then? If so, filtered replication has you
covered, right? If you just want a local mirror of the things you use,
then pull through a caching proxy (squid3 is nice).


On 27 November 2013 14:07, Jason Smith <> wrote:
> I am on-record as supportive of npm's needs here but still skeptical or at
> least unclear on the fix.
> Pain points:
> 1. Replicating the registry can have problems. Best-case scenario you have
> to download 100GB of data (and growing exponentially)
> 2. Bandwidth costs of the primary service
> Both problems are aggravated by the need for legacy support. CouchDB 1.5
> has replication fixes but it is still new. And the npm clients will want to
> fetch attachments from the couch server even after a newer CDN-using one
> comes out.
> I am conflicted. I agree with the point that you shouldn't have to
> replicate 100GB (and growing) data just to have a local npm web service.
> But I don't want to modify replication or the HTTP lightly.
> On Wed, Nov 27, 2013 at 2:14 PM, Alexander Shorin <> wrote:
>> > Move attachments out of CouchDB: Work has begun to move the package
>> tarballs out of
>> > CouchDB and into Joyent's Manta service. Additionally, MaxCDN has
>> generously offered to
>> > provide CDN services for npm, once the tarballs are moved out of the
>> registry database.
>> > This will help improve delivery speed, while dramatically reducing the
>> file system I/O load on
>> > the CouchDB servers. Work is progressing slowly, because at each stage
>> in the plan, we are
>> > making sure that current replication users are minimally impacted.
>> I wonder is it CouchDB non-optimal I/O and/or can 769 issue fix it?
>> There is alpha-patch attached. May be it's good time to push it
>> forward? What things are left for it?
>> --
>> ,,,^..^,,,

View raw message