Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <517EEA6F.3030707@zedeler.dk>
Date: Mon, 29 Apr 2013 23:47:27 +0200
From: "Michael Zedeler." <michael@zedeler.dk>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130329 Thunderbird/17.0.5
MIME-Version: 1.0
To: user@couchdb.apache.org
Subject: Re: CouchDB handling extreme loads
References: <517E45ED.90308@zedeler.dk>
 <CABvT1DFDWNjZsTm4QcNJ+HCu4N8Mbyzdi0NK=ZjupX3c0T-vzA@mail.gmail.com>
 <517EE35B.2070809@zedeler.dk>
In-Reply-To: <517EE35B.2070809@zedeler.dk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Hi Robert.

(Again.)

The cause has been found: the server ran out of memory due to a memory 
leak in my script.

Regards,

Michael.

On 2013-04-29 23:17, Michael Zedeler. wrote:
> Hi Robert.
>
> Thanks for the suggestion to use the changes feed in order to do 
> incremental backups. Haven't got any crash information, but will try 
> to generate one and post it here.
>
> Regards,
>
> Michael.
>
> On 2013-04-29 13:40, Robert Newson wrote:
>> You'd be much better off backing up by reading
>> /dbname/_changes?include_docs=true.
>>
>> Reading _all_docs and then fetching each document should work fine,
>> it'll just be much slower (and non-incremental, you'll have to start
>> from scratch every time you backup).
>>
>> Does your log include any crash information?
>>
>> B.
>>
>>
>> On 29 April 2013 11:05, Michael Zedeler. <michael@zedeler.dk> wrote:
>>> Hi.
>>>
>>> I have found a way to write a backup script using an event driven
>>> environment.
>>>
>>> For starters, I have just used the na�ve approach to get all 
>>> document ids
>>> and then fetch one at a time.
>>>
>>> This works on small databases, but for obvious reasons, the load 
>>> becomes too
>>> big on larger databases, since my script is essentially trying to 
>>> fetch too
>>> many documents at the same time.
>>>
>>> I know that I have to throttle the requests, but it turned out that 
>>> CouchDB
>>> doesn't handle the load gracefully. At some point, I just get a "Apache
>>> CouchDB starting" entry in the log and at the same time I can see 
>>> that at
>>> least one of the running requests are closed before CouchDB has 
>>> returned
>>> anything.
>>>
>>> Is this behaviour intentional? How do I send as many requests as 
>>> possible
>>> without causing the server to restart?
>>>
>>> I'd definately prefer if the server could just start responding more 
>>> slowly.
>>>
>>> I am using CouchDB 1.2 (and perls AnyEvent::CouchDB on the client - 
>>> I gave
>>> up on nano).
>>>
>>> Regards,
>>>
>>> Michael.
>>>
>