incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <>
Subject Re: chunked response and couch_doc_open
Date Fri, 23 Oct 2009 18:27:33 GMT
On Oct 23, 2009, at 2:11 PM, Norman Barker wrote:

> On Fri, Oct 23, 2009 at 11:33 AM, Paul Davis
> <> wrote:
>> On Fri, Oct 23, 2009 at 1:27 PM, Norman Barker < 
>> > wrote:
>>> Hi,
>>> is there a way (in Erlang) to open a couchdb document and to iterate
>>> over the document body without having to open up all of the document
>>> in memory?
>>> I would like to use a chunked response to keep the system having a  
>>> low
>>> memory overhead.
>>> Not a particular couch question, is there a method in erlang to find
>>> the size (as in number of bytes) of a particular term?
>>> many thanks,
>>> Norman
>> Norman,
>> Well, for document JSON we store Erlang term binaries on disk so
>> there's no real way to stream a doc across the wire from disk without
>> loading the whole thing into RAM. Have you noticed CouchDB having
>> memory issues on read loads? Its generally pretty light on its memory
>> requirements for reads.
>> The only way to get the size of a Term in bytes that I know of is the
>> brute force: size(term_to_binary(Term)) method.
>> Paul Davis
> I am sending sizeable JSON documents (a couple of mb), as this scales
> by X concurrent users then the problem grows. I have crashed erlang
> when the process gets up to about a 1gb of memory.  (Note, this was on
> windows) The workaround is to increase the memory allocation.
> Erlang (and couchdb) is fantastic in that it is so light to run as
> opposed to a J2EE server, streaming documents out would be good
> optimisation. Running a couchdb instance in < 30mb of memory space
> would be my ideal.
> If you can point me in the right direction then this is something I
> can contribute back, most of my erlang code so far has been specific
> to my application.
> Many thanks,
> Norman

Hi Norman, could your application store some of that data as an  
attachment to the document?  Attachments can be streamed in both  
directions.  Best,


View raw message