couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Nagy <>
Subject Re: Timetable for 0.10.0
Date Tue, 07 Jul 2009 09:42:26 GMT
Hi Adam,

If you could give me a full set a features needed it would be very  
useful. I feel like hacking on lhttpc a bit. The ones you listed  
already seem like interesting/useful things to add.

Just a note: chunked downloads have been recently added to lhttpc.

Imho it would be interesting to have two choices and test them against  
each other in couchdb.


Tamas Nagy
Erlang Training & Consulting

On 6 Jul 2009, at 19:02, Adam Kocoloski wrote:

> Hi Tamas, in this case I'm talking about letting the user decide  
> when to switch the socket back into {active,once} mode to receive  
> the next message.  We like this feature because it allows us to  
> incrementally process the result of a call to _changes without using  
> any memory to actually store the response (which may be 100s of MBs).
> I poked around lhttpc when ETC released it, but you're right that  
> it's still missing a number of features that we use in the  
> replicator.  Off the top of my head I can name
> * chunked downloads
> * chunked uploads
> * streaming response bodies
> Regarding binaries, our solution so far has been to make requests  
> that we can reasonably expect to have large response bodies (e.g.  
> attachment downloads) on dedicated connections outside the ibrowse  
> connection pool and garbage_collect() those processes as necessary.   
> Memory management is certainly a concern for us, and I'm curious to  
> see how the new version of ibrowse behaves now that it's using  
> binaries internally as well.
> Thanks for bringing lhttpc to our attention, but at the moment it  
> seems that only ibrowse offers the features that we would like in  
> our replicator HTTP client.  Cheers,
> Adam
> On Jul 6, 2009, at 6:00 AM, Tamas Nagy wrote:
>> Hi Adam,
>> What kind of control of the socket behaviour? lhttpc might be a  
>> good candidate as well as it is steadily building up its feature  
>> set with things which are necessary for couchdb. (like chunked).
>> Arguably ibrowse is a much mature client supporting a lot of  
>> different options (and lhttpc might not have all the required  
>> features yet), but with the recent introduction of using binaries  
>> combined with the long lived processes inside ibrowse can result in  
>> nasty memory blowups as binaries are reference counted in the VM  
>> hence the GC might not be able to get rid of the huge binaries fast  
>> enough during data transfer.
>> Regards,
>> 	Tamas
>> Tamas Nagy
>> Erlang Training & Consulting
>> On 4 Jul 2009, at 01:02, Adam Kocoloski wrote:
>>> On Jul 3, 2009, at 7:28 PM, Chris Anderson wrote:
>>>> Especially if we can get the replicator based on _changes, and  
>>>> then truly deprecate the update_notification process
>>> Chandru Mullaparthi gave us a nice assist on that front today with  
>>> an update to ibrowse that lets us control the socket behavior.  As  
>>> far as I know ibrowse is the only Erlang HTTP client that does  
>>> this correctly.  One month will be more than enough time to build  
>>> a replicator based on _changes now that this piece of the puzzle  
>>> is resolved.
>>> Adam

View raw message