Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of norman.barker@gmail.com
 designates 209.85.211.186 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KlX8nqypFAaGtX/UUuSjMWmVDdwVwBBJ+vzXQvCnp7p9uLG3AENhHWjBFfcNsZAc+I
         GTaE4AmiVKIO31qEN66anw/JuWQLStx40892151YCgUya0VJHi+Eh3txAPQn4MCTKan4
         GQfnY9Q6FoULHsnq9897aYsCzGH4FinJKiOGU=
MIME-Version: 1.0
In-Reply-To: <e2111bbb0910231119k6e2a330bv9084325008fea40@mail.gmail.com>
References: <fbbaf7b0910231027g352faf8bgcdc6a1f9c442170b@mail.gmail.com>
	 <e2111bbb0910231033r6295f7b4p552821e1691b8529@mail.gmail.com>
	 <fbbaf7b0910231111j5a81d7c8v2e75edaee59cbdc3@mail.gmail.com>
	 <e2111bbb0910231119k6e2a330bv9084325008fea40@mail.gmail.com>
Date: Fri, 23 Oct 2009 12:29:45 -0600
Message-ID: <fbbaf7b0910231129q4dd82010o4d6d4ecb3007511b@mail.gmail.com>
Subject: Re: chunked response and couch_doc_open
From: Norman Barker <norman.barker@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Oct 23, 2009 at 12:19 PM, Paul Davis
<paul.joseph.davis@gmail.com> wrote:
> On Fri, Oct 23, 2009 at 2:11 PM, Norman Barker <norman.barker@gmail.com> =
wrote:
>> On Fri, Oct 23, 2009 at 11:33 AM, Paul Davis
>> <paul.joseph.davis@gmail.com> wrote:
>>> On Fri, Oct 23, 2009 at 1:27 PM, Norman Barker <norman.barker@gmail.com=
> wrote:
>>>> Hi,
>>>>
>>>> is there a way (in Erlang) to open a couchdb document and to iterate
>>>> over the document body without having to open up all of the document
>>>> in memory?
>>>>
>>>> I would like to use a chunked response to keep the system having a low
>>>> memory overhead.
>>>>
>>>> Not a particular couch question, is there a method in erlang to find
>>>> the size (as in number of bytes) of a particular term?
>>>>
>>>> many thanks,
>>>>
>>>> Norman
>>>>
>>>
>>> Norman,
>>>
>>> Well, for document JSON we store Erlang term binaries on disk so
>>> there's no real way to stream a doc across the wire from disk without
>>> loading the whole thing into RAM. Have you noticed CouchDB having
>>> memory issues on read loads? Its generally pretty light on its memory
>>> requirements for reads.
>>>
>>> The only way to get the size of a Term in bytes that I know of is the
>>> brute force: size(term_to_binary(Term)) method.
>>>
>>> Paul Davis
>>>
>>
>> I am sending sizeable JSON documents (a couple of mb), as this scales
>> by X concurrent users then the problem grows. I have crashed erlang
>> when the process gets up to about a 1gb of memory. =A0(Note, this was on
>> windows) The workaround is to increase the memory allocation.
>>
>> Erlang (and couchdb) is fantastic in that it is so light to run as
>> opposed to a J2EE server, streaming documents out would be good
>> optimisation. Running a couchdb instance in < 30mb of memory space
>> would be my ideal.
>>
>> If you can point me in the right direction then this is something I
>> can contribute back, most of my erlang code so far has been specific
>> to my application.
>>
>> Many thanks,
>>
>> Norman
>>
>
> Norman,
>
> Streaming JSON docs in and out would require massive amounts of work
> in rewriting lots of the core of CouchDB. Right down to making the
> JSON parsers stream oriented. I'm not even sure where you'd get
> started on such an undertaking.
>
> Though there was a bug reported earlier today with Windows doing weird
> things with retaining memory for _bulk_docs calls, I wonder if there's
> a connection.
>
> Paul Davis
>
Paul,

I was thinking that perhaps this could be done at the mochijson2
level,  and wonder if on the way out if there was an iterator approach
that could be used within mochijson, but perhaps this impacts the
format of the disk storage within couchdb. Certainly it is an
optimisation, but without it does limit scalability and the premise of
running on low commodity hardware. No criticism intended, I will be
looking at this at some point.

Norman