couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: fails to allocate heap on bulk save
Date Fri, 23 Oct 2009 17:20:12 GMT
On Fri, Oct 23, 2009 at 12:55 PM, Alexander Quick <alex@alexquick.com> wrote:
> Hi,
>
> I'm having an issue where CouchDB crashes consistently while bulk
> saving(all_or_nothing is off). It completes fewer than a million documents
> of the following form before choking:
> {
>  "_id": "an_id",
>  "user_id": "a_user",
>  "attribute_one":"<7chars",
>  "attribute_two":"<7chars",
>  "attribute_three":"<7chars",
> }
>
> CouchDB 0.10 is running on Windows Server 2008 (might be my first mistake)
> and trying to insert only in 3k batches, so I don't think individual batch
> size is an issue(along with the fact that the first 30-40 batches succeed).
>
> I get the following error:
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 50152200 bytes of memory (of type "heap").
> Sometimes it fails while trying to allocate some other type of memory, but
> heap is most common.
>
> The machine still has plenty of memory left when this happens, and we're
> talking about a 50mb allocation. Even with the vm heap parameter set to 1gb,
> it chokes around 700mb-- either way this shouldn't be an issue.
>
> The only thing I can think of that separates this from an entirely
> run-of-the-mill use case is the fact that I'm inserting into approximately
> 100 databases. Though not concurrently, but rather say a batch to the first
> then a batch to the second after the first completes. Nonetheless, the vm
> keeps piling on new processes for some reason.
>
> The dump is available here: http://sandbox.stoopdev.com/fail/erl_crash.dump
> Please let me know if you have any ideas.
>
> Thanks in advance,
> Alex Quick
>

Alex,

Most odd. Can you see what the memory profile does as you do your
bulk_docs calls? Does it appear to just continuously grow and not
release RAM as you make multiple calls? It shouldn't hold onto any
memory after a _bulk_docs request is completed.

Paul Davis

Mime
View raw message