Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of nicholasredlin@gmail.com
 designates 209.85.223.175 as permitted sender)
From: Nicholas Westlake <nicholasredlin@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Limits on Continuous Replication
Message-Id: <86DF8A23-56CC-4236-9BB8-6E2536ECAC93@gmail.com>
Date: Thu, 10 Oct 2013 02:18:44 -0500
To: user@couchdb.apache.org
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))

I was advised in #couchdb that my question is better suited to the =
mailing list.

I'm looking at an app design that uses quite a few replications. I'm =
wondering what limits I'll run into from couch. Ideas for how to work =
around those limits would be bonus. :)

I have 3 kinds of database:

1) user # One of these for each user (readable and writeable only by the =
user who owns it)
2) project-public # One of these for each project (writeable only by the =
replicator, readable by everyone)
3) project-admin # One of these for each project (writeable only by the =
replicator, readable only by users granted permission)

When a user is added to a project, filtered continuous replication =
starts from their "user" database to both the "project-public" and =
"project-admin" databases. The filter functions check for simple =
conditions like "doc.committed =3D=3D=3D true". The replication will be =
stored in _replicator database so the continuous replication will =
survives a restart.

The important behaviors:

- When a user changes data and it should propagate to the the project =
databases in under 30 seconds.
- Project data should be access controlled between public data and =
admin-only data. I believe separate databases with _security set =
accordingly is the only way to achieve this.
- Restarting couch shouldn't break anything.

Some realistic maximums. It's unlikely that:

- any "project-admin" or "project-public" databases will exceed 8 =
megabytes, compacted.
- any "user" database will exceed 100 megabytes each, compacted.
- the user count, and project count will ever exceed 100,000 and 40,000 =
respectively.
- more than 4,000 users will work on any single project.
- more than 2,000 users will be making changes at any given moment.

Just in case it matters:

- The external process that will be creating databases and entries in =
the _replicator database is node.js.

The continuous replication count (at most) would be 320 million (40,000 =
projects * 4,000 users/project * 2 replications/user). This seems like =
it would break something. :)

Should this work? Or: what's the canonical (or a good) alternative to =
this design with couch?=