couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antony Blakey <>
Subject Re: Attachment Replication Problem - Bug Found
Date Sat, 16 May 2009 15:22:56 GMT

On 16/05/2009, at 11:07 PM, Adam Kocoloski wrote:

> No, I don't believe so.  ibrowse accepts a {stream_to, pid()}  
> option.  It accumulates packets until it reaches a threshold  
> configurable by {stream_chunk_size, integer()} (default 1MB), then  
> sends the data to the Pid.  I don't think ibrowse is writing to disk  
> at any point  in the process.  We do see that when streaming really  
> large attachments, ibrowse becomes the biggest memory user in the  
> emulator.

This is what I thought was happening, which means that with small  
documents with many attachments (say > 1Mb) you could potentially end  
up with masses of open connections representing data promises that are  
only forced at checkpoint time, so that's not scalable. I think the  
number of open ibrowse connections (which I see doesn't neccessariy  
match the number of unforced promises), needs to be an input to the  
checkpoint decision.

> ibrowse does offer a {save_response_to_file, boolean()|filename()}  
> option that we could possibly leverage.

That sounds like a better idea.

> Are you keeping an eye on the memory usage?  I think an out of  
> memory error can trigger this sudden death in Erlang.

Not a memory failure - it happens at the same place with either 1 or  
1.5G. Once again I have a repeatable failure that would normally be  
random :/ I'm just wondering how to debug it.

Antony Blakey
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

A priest, a minister and a rabbi walk into a bar. The bartender says  
"What is this, a joke?"

View raw message