couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cottlehuber <...@jsonified.com>
Subject Re: Having trouble replicating with a large couch instance (npm repo)
Date Tue, 14 May 2013 09:33:33 GMT
On 14 May 2013 05:30, andy e <virtualandy@gmail.com> wrote:

> Hi,
>
> Based on the awesome feedback I got from the nodejs list (see
> https://groups.google.com/d/msg/nodejs/sX4mbsRPwls/WtDDE-To2o4J), we tried
> replicating with their couch instance so we could use it in an offline
> environment.
>
> We're essentially following the instructions at
> http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository but
> replication fails after syncing about 17k documents.
>
> We've tried reinstalling couch (found one issue that suggested using a
> patched version of SpiderMonkey) but the same thing keeps happening, even
> after restarting replication several times.
>
> Here's our setup:
>
> CentOS 6.4
> CouchDB 1.3
> SpiderMonkey 1.8.5-7
>
> Replication works fine for over 17,000 documents, then we see this error
> and can't get past it:
>
> [Sat, 11 May 2013 00:55:39 GMT] [error] [<0.12970.4>] Replicator: couldn't
> write document `bufferhelper`, revision
> `19-d339684ee7f5eaf4cc18d84da753832d`, to target database `registry`.
> Error: `unauthorized`, reason: `Please log in before writing to the db`.
>
> Any ideas?
>
> Thanks,
>
> Andy
>

Hmm, weird.

curl -svX GET http://isaacs.iriscouch.com/registry/bufferhelper |
json_reformat looks ok to me, as does:

curl -svX GET
http://registry.npmjs.org/bufferhelper/-/bufferhelper-0.2.0.tgz | tar tzf -
&& echo OK

And my local npm mirror is up to date, so let's assume that the issue is at
your end for the moment.

You can replicate single (or multiple) specific documents, instead of the
whole npm repo, using this, where doc_ids is an array of doc ids you want
to replicate:

curl -svX POST http://admin:passwd@127.0.0.1:5984/_replicate -d '{"source":"
http://isaacs.iriscouch.com/registry/", "target":"mirror",
"create_target":true,  "doc_ids":["bufferhelper"]}'
-HContent-Type:application/json | json_reformat

And if that fails, can you post the output + anything of interest in
couch.log please, along with how are you setting up your replication? e.g.
curl …. as above.

Some tips:

If you're using the replicator db (via POST _replicator endpoint), I'd
suggest ensuring you have appropriate permissions in the request, e.g.

curl -svX PUT http://admin:passwd@127.0.0.1:5984/_replicator/mirror_npm -d
'{"source":"http://isaacs.iriscouch.com/registry/", "target":"mirror",
"user_ctx": {"name": "admin"}, "doc_ids":["bufferhelper"]}'
-HContent-Type:application/json


You can obviously add "create_target":true,  and "continuous": true as
needed as usual.

In your case of needing periodic updates, I'd not use continuous
replication, as the normal one will return from the API call only when
finished successfully, or failed.

You may well find that https://npmjs.org/package/npm-gh or
http://debuggable.com/posts/private-npm-modules:4e68cc7d-1ac4-42d9-995a-343dcbdd56cbis
a more suitable option.

For publishing  modules to different registries, consider doing alias
mynpm='npm --userconfig /path/to/other/npmrc --registry
https://url/for/other/registry'  from
https://github.com/isaacs/npm/issues/2593

A+
Dave

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message