Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Fri, 1 May 2009 15:55:18 -0700
From: James Marca <jmarca@translab.its.uci.edu>
To: user@couchdb.apache.org
Subject: Re: possible bug using bulk docs?
Message-ID: <20090501225518.GD22056@translab.its.uci.edu>
Mail-Followup-To: user@couchdb.apache.org
References: <20090501204134.GA12379@translab.its.uci.edu>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20090501204134.GA12379@translab.its.uci.edu>
User-Agent: Mutt/1.4.1i

As another data point, I chunked the bulk docs call into 100 docs
(from 1200) and the server crashed again on the first bulk_docs call.
I split it into chunks of 10 docs, and the loading of all 1200+ went
off without a hitch.

Also, perhaps I was wrong in saying it was an out of memory error that
killed the server.  I watched while it was processing the 100 docs, and it
just disappeared with only about 50 to 60% of RAM allocated to the
process, and my server logs said nothing about OOM process
termination.  

Cheers,
James

On Fri, May 01, 2009 at 01:41:34PM -0700, James Marca wrote:
> Hi All, 
> 
> I'm getting an out of memory type of crash when uploading lots of
> files using bulk docs, and I'm wondering whether
> this is a known issue or user error.
> 
> I set the logging to debug and this is what I see:
> 
> 
> [Fri, 01 May 2009 20:27:30 GMT] [debug] [<0.108.0>] 'POST'
> /d12_june2007/_bulk_docs {1,1}
> Headers: [{'Connection',"TE, close"},
>           {'Content-Length',"257418477"},
>           {'Host',"127.0.0.1:5984"},
>           {"Te","deflate,gzip;q=0.3"},
>           {'User-Agent',"libwww-perl/5.805"}]
> 
> [Fri, 01 May 2009 20:31:01 GMT] [info] [<0.108.0>] 127.0.0.1 - -
> 'POST' /d12_june2007/_bulk_docs 201
> 
> 
> In top I can watch the RAM usage go up and down until finally it peaks
> and the server is just gone.
> 
> That content length is from 1278 documents.  I can easily split that
> pile up into fewer documents (and that is what I am going to do) but
> I thought I'd raise the issue here as this feels like a software bug
> somewhere.  
> 
> I am running the release 0.9, on Gentoo, using the gentoo ebuild,
> x86_64 using the latest (compiled yesterday) release of erlang on
> gentoo.  
> 
> I'm more than happy to pull new code from git or svn to test things
> out too.
> 
> Regards, 
> James
> 
> -- 
> James E. Marca
> Researcher
> Institute of Transportation Studies
> AIRB Suite 4000
> University of California
> Irvine, CA 92697-3600
> jmarca@translab.its.uci.edu
> 
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.