Mailing-List: contact couchdb-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: couchdb-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of adam.kocoloski@gmail.com
 designates 74.125.44.154 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=KLI5LcxdFoz9e40iQ+O5qvhFax/JJOH/5ouh3vXGMkBR5TEjfbYDDXYvds2fgcQmvT
         ixZv1DCzXp6cTgcWpJMc89TubDayhrKc6ECK3BmereCp+00GF9TCE4adA7EO/xh+TU1s
         UhvdTEqlfHW1lR9BIX4+ItL/AmEim4IOFumKE=
Message-Id: <24E04797-1833-4479-8B95-4929DDD424A2@gmail.com>
From: Adam Kocoloski <adam.kocoloski@gmail.com>
To: couchdb-dev@incubator.apache.org
In-Reply-To: <58AB961A-A23C-46F7-9AF8-7A1181CFDDB7@gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: [jira] Updated: (COUCHDB-160) replication performance
 improvements
Date: Thu, 13 Nov 2008 22:21:57 -0500
References: <1324923413.1226429744367.JavaMail.jira@brutus>
 <9BBA8641-6BA6-49AB-BBF3-1D12F6F2726A@apache.org>
 <58AB961A-A23C-46F7-9AF8-7A1181CFDDB7@gmail.com>

On Nov 11, 2008, at 9:54 PM, Adam Kocoloski wrote:

> I tried to do some testing with large (>1MB) binary attachments this  
> afternoon and ran into a lot of stability issues at high  
> concurrency.  Quite a few of the replicator's HTTP processes (both  
> GET and POST) died gruesome deaths with session_remotly_closed errors.

Hi, just to follow up, those session_remotly_closed errors were  
unrelated to the patch I posted.  The patch I posted definitely uses  
more memory for large documents than the trunk code since it buffers  
multiple docs at once, but I'd argue that trunk's handling of these  
large documents is suboptimal anyway.  I tried replicating a DB with  
10 2.3MB documents over HTTP; trunk required well over 100 MB to get  
this done.

Perhaps there's a way to stream attachments separately during  
replication instead of using attachments=true and going through the  
whole Base64 conversion.  Regards, Adam