Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
From: Paul Okstad <pokstad@gmail.com>
Content-Type: multipart/alternative; boundary="Apple-Mail=_BA5E4F44-8E4A-4518-B035-E32FAD62DF71"
Message-Id: <8ED3123A-2470-4F56-987C-5E46EFBB6871@gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: The state of filtered replication
Date: Wed, 25 May 2016 09:39:21 -0700
References: <78765234-9DCE-4D77-8267-18342D494051@medicmobile.org> <CAA=cwqFAo2hbivrqsWkX_fFks2YCGhCN7+8GCOf-3AeZpvCsag@mail.gmail.com> <EEBA9626-15C6-4346-929D-999EDF10D63E@medicmobile.org> <CAA=cwqHRB_rgZPzJ3vttrp6t9t-r4D7Pz96mtVry7wC3cjkrzw@mail.gmail.com>
To: user@couchdb.apache.org
In-Reply-To: <CAA=cwqHRB_rgZPzJ3vttrp6t9t-r4D7Pz96mtVry7wC3cjkrzw@mail.gmail.com>
archived-at: Wed, 25 May 2016 16:51:50 -0000

--Apple-Mail=_BA5E4F44-8E4A-4518-B035-E32FAD62DF71
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

This isn=E2=80=99t just a problem of filtered replication, it=E2=80=99s =
a major issue in the database-per-user strategy (at least in the v1.6.1 =
I=E2=80=99m using). I=E2=80=99m also using a database-per-user design =
with thousands of users and a single global database. If a small =
fraction of the users (hundreds) has continuously ongoing replications =
from the user DB to the global DB, it will cause extremely high CPU =
utilization. This is without any replication filtered javascript =
function.

Another huge issue with filtered replications is that they lose their =
place when replications are restarted. In other words, they don=E2=80=99t =
keep track of sequence ID between restarts of the server or stopping and =
starting the same replication. So for example, if I want to perform =
filtered replication of public documents from the global DB to the =
public DB, and I have a ton of documents in global, then each time I =
restart the filtered replication it will begin from sequence #1. I=E2=80=99=
m guessing this is due to the fact that CouchDB does not know if the =
filter function has been modified between replications, but this =
behavior is still very disappointing.

=E2=80=94=20
Paul Okstad
http://pokstad.com <http://pokstad.com/>


> On May 25, 2016, at 4:25 AM, Stefan Klein <st.fankl.in@gmail.com> =
wrote:
>=20
> 2016-05-25 12:48 GMT+02:00 Stefan du Fresne <stefan@medicmobile.org>:
>=20
>=20
>=20
>> So to be clear, this is effectively replacing replication=E2=80=94 =
where the
>> client negotiates with the server for a collection of changes to =
download=E2=80=94
>> with a daemon that builds up a collection of documents that each =
client
>> should get (and also presumably delete), which clients can then query =
for
>> when they=E2=80=99re able?
>>=20
>=20
> Sorry, didn't describe well enough.
>=20
> On Serverside we have one big database containing all documents and =
one db
> for each user.
> The clients always replicate to and from their individual userdb,
> unfiltered. So the db for a user is a 1:1 copy of their pouchdb/... on
> their client.
>=20
> Initially we set up a filtered replication for each user from servers =
main
> database to the server copy of the users database.
> With this we ran into performance problems and sooner or later we =
probably
> would have ran into issues with open file descriptors.
>=20
> So what we do instead is listening to the changes of the main database =
and
> distribute the documents to the servers userdb, which then are synced =
with
> the clients.
>=20
> Note: this is only for documents the users actually work with (as in
> possibly modify), for queries on the data we query views on the main
> database.
>=20
> For the way back, we listen to the _dbchanges, so we get an event for
> changes on the users dbs, get that change from the users db and =
determine
> what to do with it.
> We do not replicate back users changes to the main database but rather =
have
> an internal API to evaluate all kinds of constrains on users input.
> If you do not have to check users input, you could certainly listen to
> _dbchanges and "blindly" one-shot replicate from the changed DB to =
your
> main DB.
>=20
> --=20
> Stefan


--Apple-Mail=_BA5E4F44-8E4A-4518-B035-E32FAD62DF71--