Return-Path: 
 <dev-return-18490-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org>
X-Original-To: apmail-couchdb-dev-archive@www.apache.org
Delivered-To: apmail-couchdb-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 10B1F7010
	for <apmail-couchdb-dev-archive@www.apache.org>;
 Tue, 11 Oct 2011 01:53:02 +0000 (UTC)
Received: (qmail 22785 invoked by uid 500); 11 Oct 2011 01:52:59 -0000
Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org
Received: (qmail 22728 invoked by uid 500); 11 Oct 2011 01:52:59 -0000
Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@couchdb.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@couchdb.apache.org>
List-Post: <mailto:dev@couchdb.apache.org>
List-Id: <dev.couchdb.apache.org>
Reply-To: dev@couchdb.apache.org
Delivered-To: mailing list dev@couchdb.apache.org
Received: (qmail 22714 invoked by uid 99); 11 Oct 2011 01:52:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 01:52:59 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of randall.leeds@gmail.com
 designates 209.85.210.180 as permitted sender)
Received: from [209.85.210.180] (HELO mail-iy0-f180.google.com)
 (209.85.210.180)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 01:52:53 +0000
Received: by iakc1 with SMTP id c1so6799140iak.11
        for <multiple recipients>; Mon, 10 Oct 2011 18:52:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        bh=Mii+88SkVgeGl9PPT/YrNujmylBnu5uUN340immEowY=;
        b=aX7rPNoVsDT1ymVIbgA/433PEGDaGHOUbjoquq+A0+O6jLpgKVZ+Ta+pohAdXGvDBs
         pZYBdiYhluM5I4/R468ZmhMSocvFKx26uzRqoHZgNy7U4jU+lVan4ITeHQ1G2rLpdUi7
         ed3tWkukn94hfP30xxSxVWjivto0Evs7fdFig=
MIME-Version: 1.0
Received: by 10.43.130.133 with SMTP id hm5mr21869116icc.11.1318297951391;
 Mon, 10 Oct 2011 18:52:31 -0700 (PDT)
Received: by 10.42.240.65 with HTTP; Mon, 10 Oct 2011 18:52:31 -0700 (PDT)
In-Reply-To: 
 <CALKFbxt0uBsDD3G_wjfTuOVA147Vosj_JWS9QJHft+-h1vqSZg@mail.gmail.com>
References: 
 <CALKFbxtR=m3QmXnzvEOSamQ6qRow48Sg+J1RHcoAMmwpzs4zrw@mail.gmail.com>
	<CAL3q7H4ryQJATque+A1v0kbHqfSrSoKZ8kNUaoMDR-6NPaH-Kg@mail.gmail.com>
	<CALKFbxt0uBsDD3G_wjfTuOVA147Vosj_JWS9QJHft+-h1vqSZg@mail.gmail.com>
Date: Mon, 10 Oct 2011 18:52:31 -0700
Message-ID: 
 <CAAL6JQjNO4r6P_iPa47aZbaEeePWgPsjvhuCE31-Td0WYF3zMg@mail.gmail.com>
Subject: Re: CouchDB Replication lacking resilience for many database
From: Randall Leeds <randall.leeds@gmail.com>
To: dev@couchdb.apache.org
Cc: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=20cf307f3b8a19cf4104aefc29ce

--20cf307f3b8a19cf4104aefc29ce
Content-Type: text/plain; charset=UTF-8

On Mon, Oct 10, 2011 at 17:02, Chris Stockton <chrisstocktonaz@gmail.com>wrote:

> Hello,
>
> On Mon, Oct 10, 2011 at 4:19 PM, Filipe David Manana
> <fdmanana@apache.org> wrote:
> > On Tue, Oct 11, 2011 at 12:03 AM, Chris Stockton
> > <chrisstocktonaz@gmail.com> wrote:
> > Chris,
> >
> > That said work is in the'1.2.x' branch (and master).
> > CouchDB recently migrated from SVN to GIT, see:
> > http://couchdb.apache.org/community/code.html
> >
>
> Thank you very much for the response Filipe, do you possibly have any
> documentation or more detailed summary on what these changes include
> and possible benefits of them? I would love to hear about any tweaking
> or replication tips you may have for our growth issues, perhaps you
> could answer a basic question if nothing else: Do the changes in this
> branch minimize the performance impact of continuous replication on
> many databases?
>

The primary change, as I understand it, is that CouchDB explicitly manages a
pool of HTTP connections per replication.

Previously, the pool was handled by the HTTP client module CouchDB uses,
ibrowse, which pools connections per host/port. Therefore, the pool is
shared between all the replications pulling from a given server.

The config file has settings for changing how these pools behave, under the
replicator heading:
max_http_sessions
max_http_pipeline_size

The first refers to the size of the per-host pool. The second refers to the
number of requests that can be queued for each. Unfortunately, there is one
more setting which is not exposed, which is the maximum number of requests
to try at once per replication (side note to devs, should we provide a quick
patch for this for 1.1.1?), and it is fixed at 100. CouchDB does not
elegantly handle the case when the pool is completely utilized. This is not
a problem in the new replication code in 1.2.

For the time being though, the following formula should hold true or you
will experience problems:

max_http_sessions * max_http_pipeline_size >= 100 * N

where N is the maximum number of concurrent replications triggered to pull
or push from a single host.

I believe this to still be the case in 1.1. I'm pretty sure it was at one
point earlier.

-Randall



>
> Regardless I plan on getting a build of that branch and doing some
> testing of my own very soon.
>
> Thank you!
>
> -Chris
>

--20cf307f3b8a19cf4104aefc29ce--