Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: <517EEA6F.3030707@zedeler.dk>
References: <517E45ED.90308@zedeler.dk>
	<CABvT1DFDWNjZsTm4QcNJ+HCu4N8Mbyzdi0NK=ZjupX3c0T-vzA@mail.gmail.com>
	<517EE35B.2070809@zedeler.dk>
	<517EEA6F.3030707@zedeler.dk>
Date: Mon, 29 Apr 2013 22:52:07 +0100
Message-ID: 
 <CABvT1DFTf4qMWY2+n7kPibtY+0+fMMVOVut820Qy+xC2mLnvrQ@mail.gmail.com>
Subject: Re: CouchDB handling extreme loads
From: Robert Newson <rnewson@apache.org>
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Aha! Thanks for the update.

On 29 April 2013 22:47, Michael Zedeler. <michael@zedeler.dk> wrote:
> Hi Robert.
>
> (Again.)
>
> The cause has been found: the server ran out of memory due to a memory le=
ak
> in my script.
>
> Regards,
>
> Michael.
>
>
> On 2013-04-29 23:17, Michael Zedeler. wrote:
>>
>> Hi Robert.
>>
>> Thanks for the suggestion to use the changes feed in order to do
>> incremental backups. Haven't got any crash information, but will try to
>> generate one and post it here.
>>
>> Regards,
>>
>> Michael.
>>
>> On 2013-04-29 13:40, Robert Newson wrote:
>>>
>>> You'd be much better off backing up by reading
>>> /dbname/_changes?include_docs=3Dtrue.
>>>
>>> Reading _all_docs and then fetching each document should work fine,
>>> it'll just be much slower (and non-incremental, you'll have to start
>>> from scratch every time you backup).
>>>
>>> Does your log include any crash information?
>>>
>>> B.
>>>
>>>
>>> On 29 April 2013 11:05, Michael Zedeler. <michael@zedeler.dk> wrote:
>>>>
>>>> Hi.
>>>>
>>>> I have found a way to write a backup script using an event driven
>>>> environment.
>>>>
>>>> For starters, I have just used the na=C4=ABve approach to get all docu=
ment
>>>> ids
>>>> and then fetch one at a time.
>>>>
>>>> This works on small databases, but for obvious reasons, the load becom=
es
>>>> too
>>>> big on larger databases, since my script is essentially trying to fetc=
h
>>>> too
>>>> many documents at the same time.
>>>>
>>>> I know that I have to throttle the requests, but it turned out that
>>>> CouchDB
>>>> doesn't handle the load gracefully. At some point, I just get a "Apach=
e
>>>> CouchDB starting" entry in the log and at the same time I can see that
>>>> at
>>>> least one of the running requests are closed before CouchDB has return=
ed
>>>> anything.
>>>>
>>>> Is this behaviour intentional? How do I send as many requests as
>>>> possible
>>>> without causing the server to restart?
>>>>
>>>> I'd definately prefer if the server could just start responding more
>>>> slowly.
>>>>
>>>> I am using CouchDB 1.2 (and perls AnyEvent::CouchDB on the client - I
>>>> gave
>>>> up on nano).
>>>>
>>>> Regards,
>>>>
>>>> Michael.
>>>>
>>
>