Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of jason.h.smith@gmail.com
 designates 209.85.214.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABvT1DHYRJXJVZf+abdAPTnOQbGaiDQppAczkXDJyRbA61_7zA@mail.gmail.com>
References: 
 <CAHdjipKokz6h1DhF-1uGcxRisPRmrMM7yXSAEneUSmodizc3ag@mail.gmail.com>
 <CACLy94XULKYXLZ0HMDFfzcJuPavhvqu4ET63o1HhObqBeS+W1w@mail.gmail.com>
 <CABvT1DHYRJXJVZf+abdAPTnOQbGaiDQppAczkXDJyRbA61_7zA@mail.gmail.com>
From: Jason Smith <jason.h.smith@gmail.com>
Date: Wed, 27 Nov 2013 21:18:13 +0700
Message-ID: 
 <CACLy94Xh1TbOFYdnu=490PJ5vmbuLsXXGf85szuyTYditS5A2w@mail.gmail.com>
Subject: Re: NPM, CouchDB and big attachments
To: "dev@couchdb.apache.org" <dev@couchdb.apache.org>
Content-Type: multipart/alternative; boundary=089e0141a79cafd38e04ec29456c

--089e0141a79cafd38e04ec29456c
Content-Type: text/plain; charset=UTF-8

Yes, I think it is quite likely that an alternative replicator comes out
soon. This replicator can be tailored to npm's needs, such as only
replicating a subset of the documents[1] ("Replicate only package X, Y, and
Z plus their dependencies). Or possibly even only replicating a subset of
that document's attachments. Who knows?

[1]: Yes replication filters can do that, but the npm filter policy depends
on relationships between different documents, so it's difficult to shoehorn
that into the current filter system. Perhaps the current filter will be
used as a component in a broader tool.


On Wed, Nov 27, 2013 at 9:14 PM, Robert Newson <rnewson@apache.org> wrote:

> "replicate 100GB (and growing) data just to have a local npm web service."
>
> This makes no sense to me. Of course you have to have a local copy of
> all the things to have a "local npm web service". Are we talking about
> partial mirroring instead then? If so, filtered replication has you
> covered, right? If you just want a local mirror of the things you use,
> then pull through a caching proxy (squid3 is nice).
>
> B.
>
>
> On 27 November 2013 14:07, Jason Smith <jhs@apache.org> wrote:
> > I am on-record as supportive of npm's needs here but still skeptical or
> at
> > least unclear on the fix.
> >
> > Pain points:
> >
> > 1. Replicating the registry can have problems. Best-case scenario you
> have
> > to download 100GB of data (and growing exponentially)
> >
> > 2. Bandwidth costs of the primary service
> >
> > Both problems are aggravated by the need for legacy support. CouchDB 1.5
> > has replication fixes but it is still new. And the npm clients will want
> to
> > fetch attachments from the couch server even after a newer CDN-using one
> > comes out.
> >
> > I am conflicted. I agree with the point that you shouldn't have to
> > replicate 100GB (and growing) data just to have a local npm web service.
> > But I don't want to modify replication or the HTTP lightly.
> >
> > On Wed, Nov 27, 2013 at 2:14 PM, Alexander Shorin <kxepal@gmail.com>
> wrote:
> >
> >> http://blog.nodejs.org/2013/11/26/npm-post-mortem/
> >>
> >> > Move attachments out of CouchDB: Work has begun to move the package
> >> tarballs out of
> >> > CouchDB and into Joyent's Manta service. Additionally, MaxCDN has
> >> generously offered to
> >> > provide CDN services for npm, once the tarballs are moved out of the
> >> registry database.
> >> > This will help improve delivery speed, while dramatically reducing the
> >> file system I/O load on
> >> > the CouchDB servers. Work is progressing slowly, because at each stage
> >> in the plan, we are
> >> > making sure that current replication users are minimally impacted.
> >>
> >> I wonder is it CouchDB non-optimal I/O and/or can 769 issue fix it?
> >>
> >> https://issues.apache.org/jira/browse/COUCHDB-769
> >>
> >> There is alpha-patch attached. May be it's good time to push it
> >> forward? What things are left for it?
> >>
> >> --
> >> ,,,^..^,,,
> >>
>

--089e0141a79cafd38e04ec29456c--