couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Attachment Replication Problem - Bug Found
Date Sun, 17 May 2009 12:46:45 GMT
On May 16, 2009, at 8:53 PM, Antony Blakey wrote:

> On 17/05/2009, at 12:09 AM, Adam Kocoloski wrote:
>
>> So, I think there's still some confusion here.  By "open  
>> connections" do you mean TCP connections to the source?  That  
>> number is never higher than 10.  ibrowse does pipeline requests on  
>> those 10 connections, so there could be as many as 1000  
>> simultaneous HTTP requests.  However, those requests complete as  
>> soon as the data reaches the ibrowse client process, so in fact the  
>> number of outstanding request during replication is usually very  
>> small.  We're not doing flow control at the TCP socket layer.
>
> IIUC, given that no attachments bodies are consumed by the  
> replicator until the documents are checkpointed, it's possible for  
> the replicator to block if the number of pending attachments in a  
> checkpoint buffer is greater than the ibrowse concurrent request  
> limit. In a case like mine, with many attachments on very small  
> documents, this is very likely. Or am I still confused? :/

There's one key point that you're overlooking.  From ibrowse'  
perspective, _there is no checkpoint buffer_.  ibrowse gets a request  
to download an attachment, and it immediately starts that request,  
sends the data in 1MB chunks to an attachment receiver process, and  
completes the request.  In theory, we could have 10,000 attachment  
receiver processes holding binaries to be written to disk, and ibrowse  
would be none the wiser.

Best, Adam

Mime
View raw message