couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benoit Chesneau <bchesn...@gmail.com>
Subject Re: NPM, CouchDB and big attachments
Date Wed, 27 Nov 2013 21:54:29 GMT
On Wed, Nov 27, 2013 at 3:07 PM, Jason Smith <jhs@apache.org> wrote:

> I am on-record as supportive of npm's needs here but still skeptical or at
> least unclear on the fix.
>
> Pain points:
>
> 1. Replicating the registry can have problems. Best-case scenario you have
> to download 100GB of data (and growing exponentially)
>
> 2. Bandwidth costs of the primary service
>
> Both problems are aggravated by the need for legacy support. CouchDB 1.5
> has replication fixes but it is still new. And the npm clients will want to
> fetch attachments from the couch server even after a newer CDN-using one
> comes ou

I guess you already tried that but wouldm't be a geolocalised gateway
rewritting URLS + a cdn would't solve that temporarely?


>
> I am conflicted. I agree with the point that you shouldn't have to
> replicate 100GB (and growing) data just to have a local npm web service.
> But I don't want to modify replication or the HTTP lightly.
>

What if storing attachements on first nodes would post them directly to the
CDN, It could take care if the attachment is already present and just
record it in the db. same config deployed on other nodes and the only you
would have to do is to replicate the db?

>
> On Wed, Nov 27, 2013 at 2:14 PM, Alexander Shorin <kxepal@gmail.com>
> wrote:
>
> > http://blog.nodejs.org/2013/11/26/npm-post-mortem/
> >
> > > Move attachments out of CouchDB: Work has begun to move the package
> > tarballs out of
> > > CouchDB and into Joyent's Manta service. Additionally, MaxCDN has
> > generously offered to
> > > provide CDN services for npm, once the tarballs are moved out of the
> > registry database.
> > > This will help improve delivery speed, while dramatically reducing the
> > file system I/O load on
> > > the CouchDB servers. Work is progressing slowly, because at each stage
> > in the plan, we are
> > > making sure that current replication users are minimally impacted.
> >
> > I wonder is it CouchDB non-optimal I/O and/or can 769 issue fix it?
> >
> > https://issues.apache.org/jira/browse/COUCHDB-769
> >
> > There is alpha-patch attached. May be it's good time to push it
> > forward? What things are left for it?
> >
> > --
> > ,,,^..^,,,
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message