couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Westlake <nicholasred...@gmail.com>
Subject Limits on Continuous Replication
Date Thu, 10 Oct 2013 07:18:44 GMT
I was advised in #couchdb that my question is better suited to the mailing list.

I'm looking at an app design that uses quite a few replications. I'm wondering what limits
I'll run into from couch. Ideas for how to work around those limits would be bonus. :)

I have 3 kinds of database:

1) user # One of these for each user (readable and writeable only by the user who owns it)
2) project-public # One of these for each project (writeable only by the replicator, readable
by everyone)
3) project-admin # One of these for each project (writeable only by the replicator, readable
only by users granted permission)

When a user is added to a project, filtered continuous replication starts from their "user"
database to both the "project-public" and "project-admin" databases. The filter functions
check for simple conditions like "doc.committed === true". The replication will be stored
in _replicator database so the continuous replication will survives a restart.

The important behaviors:

- When a user changes data and it should propagate to the the project databases in under 30
seconds.
- Project data should be access controlled between public data and admin-only data. I believe
separate databases with _security set accordingly is the only way to achieve this.
- Restarting couch shouldn't break anything.

Some realistic maximums. It's unlikely that:

- any "project-admin" or "project-public" databases will exceed 8 megabytes, compacted.
- any "user" database will exceed 100 megabytes each, compacted.
- the user count, and project count will ever exceed 100,000 and 40,000 respectively.
- more than 4,000 users will work on any single project.
- more than 2,000 users will be making changes at any given moment.

Just in case it matters:

- The external process that will be creating databases and entries in the _replicator database
is node.js.

The continuous replication count (at most) would be 320 million (40,000 projects * 4,000 users/project
* 2 replications/user). This seems like it would break something. :)

Should this work? Or: what's the canonical (or a good) alternative to this design with couch?
Mime
View raw message