couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Quick <a...@alexquick.com>
Subject Re: fails to allocate heap on bulk save
Date Fri, 23 Oct 2009 17:30:18 GMT
Paul,

Yeah, memory usage just grows and grows.
Do you think this might just be a problem with the windows version of  
CouchDB or Erlang?
I couldn't find anything online about a memory leak in this sort of  
case for either project.

Alex

On Oct 23, 2009, at 1:20 PM, Paul Davis wrote:

> On Fri, Oct 23, 2009 at 12:55 PM, Alexander Quick  
> <alex@alexquick.com> wrote:
>> Hi,
>>
>> I'm having an issue where CouchDB crashes consistently while bulk
>> saving(all_or_nothing is off). It completes fewer than a million  
>> documents
>> of the following form before choking:
>> {
>>  "_id": "an_id",
>>  "user_id": "a_user",
>>  "attribute_one":"<7chars",
>>  "attribute_two":"<7chars",
>>  "attribute_three":"<7chars",
>> }
>>
>> CouchDB 0.10 is running on Windows Server 2008 (might be my first  
>> mistake)
>> and trying to insert only in 3k batches, so I don't think  
>> individual batch
>> size is an issue(along with the fact that the first 30-40 batches  
>> succeed).
>>
>> I get the following error:
>> Crash dump was written to: erl_crash.dump
>> eheap_alloc: Cannot allocate 50152200 bytes of memory (of type  
>> "heap").
>> Sometimes it fails while trying to allocate some other type of  
>> memory, but
>> heap is most common.
>>
>> The machine still has plenty of memory left when this happens, and  
>> we're
>> talking about a 50mb allocation. Even with the vm heap parameter  
>> set to 1gb,
>> it chokes around 700mb-- either way this shouldn't be an issue.
>>
>> The only thing I can think of that separates this from an entirely
>> run-of-the-mill use case is the fact that I'm inserting into  
>> approximately
>> 100 databases. Though not concurrently, but rather say a batch to  
>> the first
>> then a batch to the second after the first completes. Nonetheless,  
>> the vm
>> keeps piling on new processes for some reason.
>>
>> The dump is available here: http://sandbox.stoopdev.com/fail/erl_crash.dump
>> Please let me know if you have any ideas.
>>
>> Thanks in advance,
>> Alex Quick
>>
>
> Alex,
>
> Most odd. Can you see what the memory profile does as you do your
> bulk_docs calls? Does it appear to just continuously grow and not
> release RAM as you make multiple calls? It shouldn't hold onto any
> memory after a _bulk_docs request is completed.
>
> Paul Davis


Mime
View raw message