couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Barker <>
Subject Re: chunked response and couch_doc_open
Date Fri, 23 Oct 2009 18:31:10 GMT
On Fri, Oct 23, 2009 at 12:27 PM, Adam Kocoloski <> wrote:
> On Oct 23, 2009, at 2:11 PM, Norman Barker wrote:
>> On Fri, Oct 23, 2009 at 11:33 AM, Paul Davis
>> <> wrote:
>>> On Fri, Oct 23, 2009 at 1:27 PM, Norman Barker <>
>>> wrote:
>>>> Hi,
>>>> is there a way (in Erlang) to open a couchdb document and to iterate
>>>> over the document body without having to open up all of the document
>>>> in memory?
>>>> I would like to use a chunked response to keep the system having a low
>>>> memory overhead.
>>>> Not a particular couch question, is there a method in erlang to find
>>>> the size (as in number of bytes) of a particular term?
>>>> many thanks,
>>>> Norman
>>> Norman,
>>> Well, for document JSON we store Erlang term binaries on disk so
>>> there's no real way to stream a doc across the wire from disk without
>>> loading the whole thing into RAM. Have you noticed CouchDB having
>>> memory issues on read loads? Its generally pretty light on its memory
>>> requirements for reads.
>>> The only way to get the size of a Term in bytes that I know of is the
>>> brute force: size(term_to_binary(Term)) method.
>>> Paul Davis
>> I am sending sizeable JSON documents (a couple of mb), as this scales
>> by X concurrent users then the problem grows. I have crashed erlang
>> when the process gets up to about a 1gb of memory.  (Note, this was on
>> windows) The workaround is to increase the memory allocation.
>> Erlang (and couchdb) is fantastic in that it is so light to run as
>> opposed to a J2EE server, streaming documents out would be good
>> optimisation. Running a couchdb instance in < 30mb of memory space
>> would be my ideal.
>> If you can point me in the right direction then this is something I
>> can contribute back, most of my erlang code so far has been specific
>> to my application.
>> Many thanks,
>> Norman
> Hi Norman, could your application store some of that data as an attachment
> to the document?  Attachments can be streamed in both directions.  Best,
> Adam

Hi Adam,

it was seeing that attachments could be streamed that got me thinking
about the JSON.  Unfortunately I couldn't use attachments since I am
storing metadata, all of which has to be view-able.


View raw message