Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of adam.kocoloski@gmail.com
 designates 74.125.92.24 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=wNiOwwViSEN8QPcofJqP8OH/OeDBCjOolhmnq6VlouGw1JraIgTbpvCsS05bXlBn0Y
         8XEOVip1u+FQXjUHAWEIpCNJaHjQ4TYkNaf2VDWlm7lyX88xjICwE2gHqiuVaA5IcOLP
         2bJuPOR081+RF1kcaUpd0WmbzGMKKUlMYmaAQ=
Sender: Adam Kocoloski <adam.kocoloski@gmail.com>
Message-Id: <5E34D6F6-51D1-43A3-96DA-8CAC00883A56@apache.org>
From: Adam Kocoloski <kocolosk@apache.org>
To: dev@couchdb.apache.org
In-Reply-To: <C9EA0649-0B44-4928-B918-2B7D48DEB934@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: replication using _changes API
Date: Fri, 12 Jun 2009 08:59:42 -0400
References: <56A7CF26-8B1D-4D98-A122-B5A77A55F337@apache.org>
 <C9EA0649-0B44-4928-B918-2B7D48DEB934@apache.org>

Hi Damien, I'm not sure I follow.  My worry was that, if I built a  
replicator which only queried _changes to get the list of updates, I'd  
have to be prepared to process a very large response.  I thought one  
smart way to process this response was to throttle the download at the  
TCP level by putting the socket into passive mode.

I agree that the HTTP client seems to be at fault, because the option  
that it exposes to switch to passive mode seems to be a no-op.  What  
exactly did you mean by "streams the data while not buffering the  
data"?  Best,

Adam

On Jun 12, 2009, at 8:03 AM, Damien Katz wrote:

> I don't think this is TCPs fault, it's the HTTP client. We need a  
> HTTP client that streams data while not buffering the data (low  
> level TCP already buffers some), instead of sending all the data  
> that comes in to the waiting process, essentially buffering  
> everything.
>
> -Damien
>
>
> On Jun 11, 2009, at 4:14 PM, Adam Kocoloski wrote:
>
>> I had some time to work on a replicator that queries _changes  
>> instead of _all_docs_by_seq today.  The first question that came to  
>> my mind was how to put a spigot on the firehose.  If I call  
>> _changes without a "since" qs parameter on a 10M document DB I'm  
>> going to get 10M chunks of output back.
>>
>> I thought I might be able to control the flow at the TCP socket  
>> level using the inets HTTP client's {stream,{self,once}} option.  I  
>> still think this would be an elegant option if I can get it to  
>> work, but my early tests show that all the chunks still show up  
>> immediately in the calling process regardless of whether I stream  
>> to self or {self,once}.
>>
>> All for now, Adam
>