couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <>
Subject Re: [DISCUSS] : things we need to solve/decide : changes feed
Date Mon, 04 Feb 2019 20:10:52 GMT
Probably good to take a quick step back and note that FoundationDB’s versionstamps are an
elegant and scalable solution to atomically maintaining the index of documents in the order
in which they were most recently updated. I think that’s what you mean by the first part
of the problem, but I want to make sure that on the ML here we collectively understand that
FoundationDB actually nails this hard part of the problem *really* well.

When you say “notify CouchDB about new updates”, are you referring to the feed=longpoll
or feed=continuous options to the _changes API? I guess I see  three different routes that
can be taken here.

One route is to use the same kind machinery that we have in place today in CouchDB 2.x. As
a reminder, the way this works is

-  a client waiting for changes on a DB spawns one local process and also a rexi RPC process
on each node hosting one of the DB shards of interest (see fabric_db_update_listener). 
- those RPC processes register as local couch_event listeners, where they receive {db_updated,
ShardName} messages forwarded to them from the couch_db_updater processes.

Of course, in the FoundationDB design we don’t need to serialize updates in couch_db_updater
processes, but individual writers could just as easily fire off those db_updated messages.
This design is already heavily optimized for large numbers of listeners on large numbers of
databases. The downside that I can see is it means the *CouchDB layer nodes would need to
form a distributed Erlang cluster* in order for them to learn about the changes being committed
from other nodes in the cluster.

So let’s say we *didn’t* want to do that and we rather are trying to design for completely
independent layer nodes that have no knowledge of or communication with one another save through
FoundationDB. There’s definitely something to be said for that constraint. One very simple
approach might be to just poll FoundationDB. If the database is under a heavy write load there’s
no overhead to this approach; every time we finish sending the output of one range query against
the versionstamp space and we re-acquire a new read version there will be new updates to consume.
Where it gets inefficient is if we have a lot of listeners on the feed and a very low-throughput
database. But one fiddle with polling intervals, or else have a layer of indirection so only
one process on each layer node is doing the polling and then sending events to couch_event.
I think this could scale quite far.

The other option (which I think is the one you’re homing in on) is to leverage FoundationDB’s
watchers to get a push notification about updates to a particular key. I would be cautious
about creating a specific key or set of keys specifically for this purpose, but, if we find
that there’s some other bit of metadata that we are needing to maintain anyway then this
could work nicely. I think same indirection that I described above (where each layer node
has a maximum of one watcher per database, and it re-broadcasts messages to all interested
clients via couch_event) would help us not be too constrained by the limit on watches.

So to recap, the three approaches

1. Writers publish db_updated events to couch_event, listeners use distributed Erlang to subscribe
to all nodes
2. Poll the _changes subspace, scale by nominating a specific process per node to do the polling
3. Same as #2 but using a watch on DB metadata that changes with every update instead of polling


> On Feb 4, 2019, at 2:18 PM, Ilya Khlopotov <> wrote:
> One of the features of CouchDB, which doesn't map cleanly into FoudationDB is changes
feed. The essence of the feature is: 
> - Subscriber of the feed wants to receive notifications when database is updated. 
> - The notification includes update_seq for the database and list of changes which happen
at that time. 
> - The change itself includes docid and rev. 
> Hi, 
> There are multiple ways to easily solve this problem. Designing a scalable way to do
it is way harder.  
> There are at least two parts to this problem:
> - how to structure secondary indexes so we can provide what we need in notification event
> - how to notify CouchDB about new updates
> For the second part of the problem we could setup a watcher on one of the keys we have
to update on every transaction. For example the key which tracks the database_size or key
which tracks the number of documents or we can add our own key. The problem is at some point
we would hit a capacity limit for atomic updates of a single key (FoundationDB doesn't redistribute
the load among servers on per key basis). In such case we would have to distribute the counter
among multiple keys to allow FoundationDB to split the hot range. Therefore, we would have
to setup multiple watches. FoundationDB has a limit on the number of watches the client can
setup (100000 watches). So we need to keep in mind this number when designing the feature.

> The single key update rate problem is very theoretical and we might ignore it for the
PoC version. Then we can measure the impact and change design accordingly. The reason I decided
to bring it up is to see maybe someone has a simple solution to avoid the bottleneck. 
> best regards,
> iilyak

View raw message