Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: <A1D52A21-7E1A-4763-AD66-503568E832BB@gmail.com>
References: <A1D52A21-7E1A-4763-AD66-503568E832BB@gmail.com>
From: Stefan Klein <st.fankl.in@gmail.com>
Date: Fri, 17 Jun 2016 12:26:33 +0200
Message-ID: <CAA=cwqF1uWivnZGBQ9wruBNrWj6Cx9WN_0mZ4q284-cb4-XLGQ@mail.gmail.com>
Subject: Re: Keep the constant size of database
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=001a113d0c86f9ce6f053576c99b
archived-at: Fri, 17 Jun 2016 10:26:41 -0000

--001a113d0c86f9ce6f053576c99b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

2016-06-17 11:26 GMT+02:00 Anatoly Smolyaninov <zarkonesmall@gmail.com>:

> Hello!
>
> I=E2=80=99m gathering lots of data with couchdb for statistics. Map-reduc=
e view
> approch gives the ability to quickly get pre-calculated data, and this is
> very handy since the data consumer software wants the data quickly and ve=
ry
> often in spite of very high speed of new incoming stat metrics.
>
> But I=E2=80=99m having troubles keeping the database always the same size=
: I need
> only data for e.g. last 3 hours and all which is elder to be deleted.
>
> I have a special view which emits the timestamp. I query that view with
> startkey-endkey params equal 3 hours ago and then bulk_update this items
> back with _deleted field. But what I'm currently see is that when I do
> delete this way & cleanup after that operation, the number of documents i=
s
> reduced as I expected but size of db has actually increased.
>
> So, after some time of this constant auto clean-up, I see that size of
> database increased very much, but number of documents remains constant.
>
> It would be kind of tricky to create a few databases and delete it
> completely, because data is constantly coming with a high rate. Another
> words, I always need that 3 hour history.
>
> Where can I read how this mechanism actually works and how should I delet=
e
> the data properly? Is it actually possible?
>


You got 2 possibilities and probably want to use both.

First is compaction.
Manual:
http://docs.couchdb.org/en/1.6.1/api/database/compact.html
Automatic:
http://docs.couchdb.org/en/1.6.1/config/compaction.html

This will still leave the deleted documents (not their body) in the DB, so
your db size will still slightly increase over time.

To solve this, you would have to create a new DB, replicate the still
needed data to this new db without replicating the deleted document.
So either a filter function to not replicate the deleted or in your case
even just documents younger than 3 hours.
Be aware, views are not replicated, so the initial view access will take
longer.

regards,
Stefan

--001a113d0c86f9ce6f053576c99b--