couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <>
Subject Re: progress update on OTP replicator
Date Sat, 07 Mar 2009 15:24:01 GMT
On Sat, Mar 7, 2009 at 10:18 AM, Adam Kocoloski <> wrote:
> Hi again, I put the new replicator through its paces over the past few days
> and am pretty happy with it.  Summary of the changes:
> * Only one copy of a given source->target replication can run at any given
> time.  Other POSTs to _replicate with the same request will be added as
> listeners to the ongoing replication.  When that replication finishes, we
> respond to the original requester and check for new updates to include in
> the response to the others.  This is Option 3 in the previous thread on dev@
> * Replications that terminate with an abnormal reason (other than shutdown)
> will be restarted by the supervisor, but the listeners on the original
> request are just given an {error, reason} response.  They can track the
> results of the retry by re-POSTing.
> * Attachments are streamed directly to disk in the case of a pull
> replication.  This feature was made possible by some bleeding-edge updates
> to ibrowse courtesy of Chandru Mullaparthi.  Is still requires quite a lot
> of memory in the ibrowse processes, but we're working on that.  The memory
> footprint in the CouchDB processes is negligible.  Testing between two EC2
> nodes indicated that pulling 400MB of attachments was approximately 20x
> faster than pushing them (push still inlines in the JSON).
> * Memory utilization has decreased.  All the tests posted by Jeff Hinrichs
> in COUCHDB-270 pass, although the tests with 20MB JSON documents still use
> over a gig of memory in ibrowse processes.
> * The parallel asynchronous GET requests we used in pull replications have
> been temporarily rolled back out.  I have a branch that includes them, but I
> want to make sure there are no regressions in the memory utilization before
> merging that feature back in.  As a result, you'll find that push
> replications will be significantly faster than pulls if there are no large
> attachments hanging around.
> * Replication updates show up in the Futon status window.
> Side note: I redefined couch_util:should_flush() so that it considers both
> the process' memory usage as well as any associated reference-counted
> binaries in calculating its response.  The code to find those binaries
> relies on a part of process_info() that "may be changed or removed without
> prior notice" and isn't terribly well documented, but we ignore binary
> memory usage at our peril.  With the old definition should_flush() would
> come back false even when the replicator had grabbed 2GB worth of binaries
> off the disk!
> If there are no objections I'd like to add this code to trunk later today.
>  Best,
> Adam


View raw message