Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
Mime-Version: 1.0 (Apple Message framework v1076)
Subject: Re: fails to allocate heap on bulk save
From: Alexander Quick <alex@alexquick.com>
In-Reply-To: <e2111bbb0910231020i7cb08b34ra2c1f5e62dce7330@mail.gmail.com>
Date: Fri, 23 Oct 2009 13:30:18 -0400
Content-Transfer-Encoding: 7bit
Message-Id: <83177F1D-627E-43DF-97EC-2CED715C1AE2@alexquick.com>
References: <7CBE5392-08B2-4D41-9F9F-95A6ABAB7945@alexquick.com>
 <e2111bbb0910231020i7cb08b34ra2c1f5e62dce7330@mail.gmail.com>
To: user@couchdb.apache.org

Paul,

Yeah, memory usage just grows and grows.
Do you think this might just be a problem with the windows version of  
CouchDB or Erlang?
I couldn't find anything online about a memory leak in this sort of  
case for either project.

Alex

On Oct 23, 2009, at 1:20 PM, Paul Davis wrote:

> On Fri, Oct 23, 2009 at 12:55 PM, Alexander Quick  
> <alex@alexquick.com> wrote:
>> Hi,
>>
>> I'm having an issue where CouchDB crashes consistently while bulk
>> saving(all_or_nothing is off). It completes fewer than a million  
>> documents
>> of the following form before choking:
>> {
>>  "_id": "an_id",
>>  "user_id": "a_user",
>>  "attribute_one":"<7chars",
>>  "attribute_two":"<7chars",
>>  "attribute_three":"<7chars",
>> }
>>
>> CouchDB 0.10 is running on Windows Server 2008 (might be my first  
>> mistake)
>> and trying to insert only in 3k batches, so I don't think  
>> individual batch
>> size is an issue(along with the fact that the first 30-40 batches  
>> succeed).
>>
>> I get the following error:
>> Crash dump was written to: erl_crash.dump
>> eheap_alloc: Cannot allocate 50152200 bytes of memory (of type  
>> "heap").
>> Sometimes it fails while trying to allocate some other type of  
>> memory, but
>> heap is most common.
>>
>> The machine still has plenty of memory left when this happens, and  
>> we're
>> talking about a 50mb allocation. Even with the vm heap parameter  
>> set to 1gb,
>> it chokes around 700mb-- either way this shouldn't be an issue.
>>
>> The only thing I can think of that separates this from an entirely
>> run-of-the-mill use case is the fact that I'm inserting into  
>> approximately
>> 100 databases. Though not concurrently, but rather say a batch to  
>> the first
>> then a batch to the second after the first completes. Nonetheless,  
>> the vm
>> keeps piling on new processes for some reason.
>>
>> The dump is available here: http://sandbox.stoopdev.com/fail/erl_crash.dump
>> Please let me know if you have any ideas.
>>
>> Thanks in advance,
>> Alex Quick
>>
>
> Alex,
>
> Most odd. Can you see what the memory profile does as you do your
> bulk_docs calls? Does it appear to just continuously grow and not
> release RAM as you make multiple calls? It shouldn't hold onto any
> memory after a _bulk_docs request is completed.
>
> Paul Davis