Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <C9EA0649-0B44-4928-B918-2B7D48DEB934@apache.org>
From: Damien Katz <damien@apache.org>
To: dev@couchdb.apache.org
In-Reply-To: <56A7CF26-8B1D-4D98-A122-B5A77A55F337@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: replication using _changes API
Date: Fri, 12 Jun 2009 08:03:04 -0400
References: <56A7CF26-8B1D-4D98-A122-B5A77A55F337@apache.org>

I don't think this is TCPs fault, it's the HTTP client. We need a HTTP  
client that streams data while not buffering the data (low level TCP  
already buffers some), instead of sending all the data that comes in  
to the waiting process, essentially buffering everything.

-Damien


On Jun 11, 2009, at 4:14 PM, Adam Kocoloski wrote:

> I had some time to work on a replicator that queries _changes  
> instead of _all_docs_by_seq today.  The first question that came to  
> my mind was how to put a spigot on the firehose.  If I call _changes  
> without a "since" qs parameter on a 10M document DB I'm going to get  
> 10M chunks of output back.
>
> I thought I might be able to control the flow at the TCP socket  
> level using the inets HTTP client's {stream,{self,once}} option.  I  
> still think this would be an elegant option if I can get it to work,  
> but my early tests show that all the chunks still show up  
> immediately in the calling process regardless of whether I stream to  
> self or {self,once}.
>
> All for now, Adam