couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <>
Subject Re: Timetable for 0.10.0
Date Mon, 06 Jul 2009 18:02:25 GMT
Hi Tamas, in this case I'm talking about letting the user decide when  
to switch the socket back into {active,once} mode to receive the next  
message.  We like this feature because it allows us to incrementally  
process the result of a call to _changes without using any memory to  
actually store the response (which may be 100s of MBs).

I poked around lhttpc when ETC released it, but you're right that it's  
still missing a number of features that we use in the replicator.  Off  
the top of my head I can name

* chunked downloads
* chunked uploads
* streaming response bodies

Regarding binaries, our solution so far has been to make requests that  
we can reasonably expect to have large response bodies (e.g.  
attachment downloads) on dedicated connections outside the ibrowse  
connection pool and garbage_collect() those processes as necessary.   
Memory management is certainly a concern for us, and I'm curious to  
see how the new version of ibrowse behaves now that it's using  
binaries internally as well.

Thanks for bringing lhttpc to our attention, but at the moment it  
seems that only ibrowse offers the features that we would like in our  
replicator HTTP client.  Cheers,


On Jul 6, 2009, at 6:00 AM, Tamas Nagy wrote:

> Hi Adam,
> What kind of control of the socket behaviour? lhttpc might be a good  
> candidate as well as it is steadily building up its feature set with  
> things which are necessary for couchdb. (like chunked).
> Arguably ibrowse is a much mature client supporting a lot of  
> different options (and lhttpc might not have all the required  
> features yet), but with the recent introduction of using binaries  
> combined with the long lived processes inside ibrowse can result in  
> nasty memory blowups as binaries are reference counted in the VM  
> hence the GC might not be able to get rid of the huge binaries fast  
> enough during data transfer.
> Regards,
> 	Tamas
> Tamas Nagy
> Erlang Training & Consulting
> On 4 Jul 2009, at 01:02, Adam Kocoloski wrote:
>> On Jul 3, 2009, at 7:28 PM, Chris Anderson wrote:
>>> Especially if we can get the replicator based on _changes, and  
>>> then truly deprecate the update_notification process
>> Chandru Mullaparthi gave us a nice assist on that front today with  
>> an update to ibrowse that lets us control the socket behavior.  As  
>> far as I know ibrowse is the only Erlang HTTP client that does this  
>> correctly.  One month will be more than enough time to build a  
>> replicator based on _changes now that this piece of the puzzle is  
>> resolved.
>> Adam

View raw message