couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Cottlehuber (JIRA)" <>
Subject [jira] [Commented] (COUCHDB-1946) Trying to replicate NPM grinds to a halt after 40GB
Date Mon, 02 Dec 2013 21:32:35 GMT


Dave Cottlehuber commented on COUCHDB-1946:

Hi Marc,

thanks for reporting this. We'll need a bit more info to make headway on this; the error message
you see is a relatively unimportant part of the Erlang VM.

If you can start your instance up, I'm interested in a couple of specific things:

what the last update sequence number of your registry copy is:

export COUCH=http://localhost:5984
curl $COUCH/registry

should return a JSON blob like this:

    "committed_update_seq": 7305,
    "compact_running": false,
    "data_size": 2040934117,
    "db_name": "registry",
    "disk_format_version": 6,
    "disk_size": 3444014974,
    "doc_count": 5208,
    "doc_del_count": 754,
    "instance_start_time": "1386014605129594",
    "purge_seq": 0,
    "update_seq": 7305

Also, if you can make the couch.log file available (privately is fine, GPG
key for optional signing we might find something more

Previous occurences of the error you mentioned have all been related to insufficient memory.
FWIW I'm running replication < 2GB RAM atm on a gce small instance, 2 cores, and that's
cpu bound only.

Current parameters can be seen  via GET /_config/replicator, and you can do updates via PUT
or also via futon's configuation interface. This will avoid the need to restart couchdb, which
is only required if you edit the ini file by hand.

worker_batch_size to ease RAM pressure
worker_processes for disk & network IO

Other Couch folk report reducing these right down has 

Bear in mind that a heavy load is borne by the central registry atm, there may not be a great
deal you can influence on your local node wrt to performance.

You can try reducing the parallelisation of replication;

> Trying to replicate NPM grinds to a halt after 40GB
> ---------------------------------------------------
>                 Key: COUCHDB-1946
>                 URL:
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Database Core
>            Reporter: Marc Trudel
> I have been able to replicate the Node.js NPM database until 40G or so, then I get this:
> I one case I have gotten a flat-out OOM error, but I didn't take a dump of the log output
at the time.
> CentOS6.4 with CouchDB 1.5 (also tried 1.3.1, but to no avail). Also tried to restart
replication from scratch - twice - bot cases stalling at 40GB.

This message was sent by Atlassian JIRA

View raw message