Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of paul.joseph.davis@gmail.com
 designates 74.125.46.30 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=n4PeBxA5hq7FE5oKnM68TN8l1nGWq5Z3AMO7lRkqQ7xnkmr7Ix08RyUnqklW6TVuqj
         0DAYfaCBnBUJY6lLk2pe+sKEQTD5/sxhJpOb6kT3G6Dd0wT5ZUlj/nSYl5046UJkHr+l
         nutTuMIkMatsZz2cv0GF55t7qnZopJSmwHDAQ=
MIME-Version: 1.0
In-Reply-To: <20090501204134.GA12379@translab.its.uci.edu>
References: <20090501204134.GA12379@translab.its.uci.edu>
Date: Fri, 1 May 2009 15:59:49 -0700
Message-ID: <e2111bbb0905011559n5bcafd37k696f0fe26ca7b77d@mail.gmail.com>
Subject: Re: possible bug using bulk docs?
From: Paul Davis <paul.joseph.davis@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

James,

This sounds like you're hitting Erlang's out of memory handling. Which
is to shutdown the entire VM. Part of the Erlang way is to fail fast
and allow the code to be restarted. It'd probably be better if we
maybe try and detect that we're running out of memory but that seems
like it could get 'interesting' real quick.

If you're doing things with attachments, you might check out the
standalone API as that should make more efficient use of memory. If
your JSON is really that big, you'll just have to tune your _bulk_docs
size to fit in RAM. And remember that there can be three copies of
your JSON in ram when you have the JSON string, the Erlang term
representation, and the binary before writing to disk.

HTH,
Paul Davis

On Fri, May 1, 2009 at 1:41 PM, James Marca <jmarca@translab.its.uci.edu> w=
rote:
> Hi All,
>
> I'm getting an out of memory type of crash when uploading lots of
> files using bulk docs, and I'm wondering whether
> this is a known issue or user error.
>
> I set the logging to debug and this is what I see:
>
>
> [Fri, 01 May 2009 20:27:30 GMT] [debug] [<0.108.0>] 'POST'
> /d12_june2007/_bulk_docs {1,1}
> Headers: [{'Connection',"TE, close"},
> =A0 =A0 =A0 =A0 =A0{'Content-Length',"257418477"},
> =A0 =A0 =A0 =A0 =A0{'Host',"127.0.0.1:5984"},
> =A0 =A0 =A0 =A0 =A0{"Te","deflate,gzip;q=3D0.3"},
> =A0 =A0 =A0 =A0 =A0{'User-Agent',"libwww-perl/5.805"}]
>
> [Fri, 01 May 2009 20:31:01 GMT] [info] [<0.108.0>] 127.0.0.1 - -
> 'POST' /d12_june2007/_bulk_docs 201
>
>
> In top I can watch the RAM usage go up and down until finally it peaks
> and the server is just gone.
>
> That content length is from 1278 documents. =A0I can easily split that
> pile up into fewer documents (and that is what I am going to do) but
> I thought I'd raise the issue here as this feels like a software bug
> somewhere.
>
> I am running the release 0.9, on Gentoo, using the gentoo ebuild,
> x86_64 using the latest (compiled yesterday) release of erlang on
> gentoo.
>
> I'm more than happy to pull new code from git or svn to test things
> out too.
>
> Regards,
> James
>
> --
> James E. Marca
> Researcher
> Institute of Transportation Studies
> AIRB Suite 4000
> University of California
> Irvine, CA 92697-3600
> jmarca@translab.its.uci.edu
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>