Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of
 tamas.nagy@erlang-consulting.com designates 88.208.247.62 as permitted
 sender)
Cc: Tamas Nagy <tamas.nagy@erlang-consulting.com>
Message-Id: <D2E3200A-CD52-4B26-A1B4-C673B9264DF8@erlang-consulting.com>
From: Tamas Nagy <tamas.nagy@erlang-consulting.com>
To: dev@couchdb.apache.org
In-Reply-To: <E109884B-6F6D-472F-BDCE-D72938D0C311@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Timetable for 0.10.0
Date: Tue, 7 Jul 2009 10:42:26 +0100
References: <3418A46D-FE98-4782-9D5A-D45DE1FD899D@apache.org>
 <e282921e0907031216q2b49c778t90c35a79e40c5b0c@mail.gmail.com>
 <20090703232147.GC7898@tumbolia.org>
 <e282921e0907031628re0d635do8eab39563e5678a1@mail.gmail.com>
 <51301D85-CDAD-4341-88EC-10D95616EE60@apache.org>
 <08DABC07-CD7E-4A4B-A297-906B94A52F4C@erlang-consulting.com>
 <E109884B-6F6D-472F-BDCE-D72938D0C311@apache.org>

Hi Adam,

If you could give me a full set a features needed it would be very  
useful. I feel like hacking on lhttpc a bit. The ones you listed  
already seem like interesting/useful things to add.

Just a note: chunked downloads have been recently added to lhttpc.

Imho it would be interesting to have two choices and test them against  
each other in couchdb.

Regards,
	Tamas

Tamas Nagy
Erlang Training & Consulting
http://www.erlang-consulting.com

On 6 Jul 2009, at 19:02, Adam Kocoloski wrote:

> Hi Tamas, in this case I'm talking about letting the user decide  
> when to switch the socket back into {active,once} mode to receive  
> the next message.  We like this feature because it allows us to  
> incrementally process the result of a call to _changes without using  
> any memory to actually store the response (which may be 100s of MBs).
>
> I poked around lhttpc when ETC released it, but you're right that  
> it's still missing a number of features that we use in the  
> replicator.  Off the top of my head I can name
>
> * chunked downloads
> * chunked uploads
> * streaming response bodies
>
> Regarding binaries, our solution so far has been to make requests  
> that we can reasonably expect to have large response bodies (e.g.  
> attachment downloads) on dedicated connections outside the ibrowse  
> connection pool and garbage_collect() those processes as necessary.   
> Memory management is certainly a concern for us, and I'm curious to  
> see how the new version of ibrowse behaves now that it's using  
> binaries internally as well.
>
> Thanks for bringing lhttpc to our attention, but at the moment it  
> seems that only ibrowse offers the features that we would like in  
> our replicator HTTP client.  Cheers,
>
> Adam
>
> On Jul 6, 2009, at 6:00 AM, Tamas Nagy wrote:
>
>> Hi Adam,
>>
>> What kind of control of the socket behaviour? lhttpc might be a  
>> good candidate as well as it is steadily building up its feature  
>> set with things which are necessary for couchdb. (like chunked).
>> Arguably ibrowse is a much mature client supporting a lot of  
>> different options (and lhttpc might not have all the required  
>> features yet), but with the recent introduction of using binaries  
>> combined with the long lived processes inside ibrowse can result in  
>> nasty memory blowups as binaries are reference counted in the VM  
>> hence the GC might not be able to get rid of the huge binaries fast  
>> enough during data transfer.
>>
>> Regards,
>> 	Tamas
>>
>> Tamas Nagy
>> Erlang Training & Consulting
>> http://www.erlang-consulting.com
>>
>> On 4 Jul 2009, at 01:02, Adam Kocoloski wrote:
>>
>>> On Jul 3, 2009, at 7:28 PM, Chris Anderson wrote:
>>>
>>>> Especially if we can get the replicator based on _changes, and  
>>>> then truly deprecate the update_notification process
>>>
>>> Chandru Mullaparthi gave us a nice assist on that front today with  
>>> an update to ibrowse that lets us control the socket behavior.  As  
>>> far as I know ibrowse is the only Erlang HTTP client that does  
>>> this correctly.  One month will be more than enough time to build  
>>> a replicator based on _changes now that this piece of the puzzle  
>>> is resolved.
>>>
>>> Adam
>>>
>>
>