couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <>
Subject Re: [DISCUSS] Per-doc access control
Date Sun, 10 Mar 2019 14:51:14 GMT
Hey all,

after mulling this over some more, I’d like to tackle the detailed API and behaviour for
this. Especially how _access work in conjunction with existing access control features.

My guiding principles so far are:

1. Make the API intuitive, things should work like they look like they should work like.
2. The default should never be that a resources is accidentally left accessible to the public.
3. This should work as a natural extension to the existing security features*.

* I’d be up for reworking the whole lot, too, but that might be a better discussion for
> 4.0.

## Database Creation and Default Behaviours

Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true

In a 3.0 world where this would land, we already agreed that databases should be admin-only
by default (instead of world read/writeable today). This is a sensible default, but that leaves
us with an _access enabled database that can’t be used by anyone by server or db admins.
Not very useful.

To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e.
if a user or a group a user belongs to is mentioned in either `admins` or `members` inside
of _security, they can proceed and create documents on the db. This puts a second step burden
on the application developer, but it slots cleanly into the existing security mechanisms,
and doesn’t require special case handling. Alternatively, we could define that _security
isn’t available in _access enabled databases, but that’s something I’d like to avoid
if at all possible.

In order to make it easy to specify that “everyone in _users” should be able to use the
db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone
in /_users” (this only excludes server admins which have full access anyway).

* * *

## Document Creation and Access Control

Next, one of our non-admin users creates a doc. There are multiple options as to how we store
the _access information.

1. Automatically translate the of a doc creation (not an update) into the first
element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}.
This is a little bit counter-intuitive.

2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting
of access permissions on doc creation and I think is preferable.

This leaves the edge case of docs that have no _access member: so far I thought those docs
are admin-only, with maybe a db-wide option to swap the default to public access, but I think
given the explicitness of 2. we can do better: require _access for all new doc creations in
access-enabled databases. A user can not create a new document without an _access field that
is an array that has at least one member. For public documents, we could invent a new role
_public, and admin-only docs could use the existing role _admin.

The one downside to this approach is that we won’t be able to replicate existing databases
into an access-enabled database without modifying all documents. This might be a worthwhile
trade-off, but we should make that decision consciously and document it well. We could allow
for a special case where an _admin user can create docs that have no _access field, and those
docs are treated as having only the _admin role in _access. So at least we could replicate
all data in, but then require a manual step to update all docs to say, migrate an existing
db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.

For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or
role in its _access field.

For both creations and updates, a user MUST supply at least one role they belong to or their
own username.

* * *

## _revs_diff

/db/_revs_diff can answer the question of which revisions of a document do NOT exist on a
replication target:

This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore),
so the result schema should be expanded to handle id: unauthorized or somesuch, something
the replicator needs to know what to do with, if it encounters it (say a user got removed
from the _access list inbetween the replicator opening _changes and requesting the doc).

The _revs_diff implementation would have to altered to send an unauthorized token for each
doc the requesting userCtx has no access to. If we can re-use some of our existing indexes,
or any other performance optimisation, that’d be great. I haven’t looked at that code
at all, yet.

An important side-effect of this is, once a user has been added to a doc’s _access list,
they get access to “the full history of the doc”, even before they had access. Of course,
in CouchDB this means only getting access to the rev ids, and not the content, but since they
are content-addressable hashes, a user could brute-force themselves into revealing certain
real values from earlier incarnations of the doc. I’d rather not track _access per document
revision in perpetuity, so this is something we have to be very up-front about.

* * *

## Partitioned Databases

I mentioned partitioned databases in my previous mail, and I think it is something we can
document that end-users can opt into, but doesn’t require any special casing on the _access
proposal. That is, if users start prefixing their doc ids with a user name or id and enable
both _access and partitions, then they get all the benefits of a partitioned database, and
if they choose not to, they don’t, but things keep working. There are enough use-cases to
warrant both behaviours.

* * *

## Scenarios that _access should help with.

Overall, we developed _access to allow users to stop using the db-per-user architecture, but
once we have per-doc-access control, folks might start using this for all manner of things.
We should be clear about which scenarios we support and which we don’t.

### Scenario 1: db-per-user

In this scenario, _access enabled databases, the only way to allow mutually untrusting users
to store data in a part of CouchDB that only they (and admins) have access to was giving each
user their own database.

In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs
knowing no other user (aside from admins) can access those docs.

This is the simplest scenario, as all we’d have to track the owner of a document and produce
by-access-id/seq indexes based on that owner.

The current prototype implementation mostly reflects this stage. Not saying this is what we
should ship, but it is the easiest do implement and explain.

Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks
who don’t need anything else.

### Scenario 2: db-per-user + Sharing

The second we allow per doc auth, users will want to share those docs with other users. That’s
why we initially suggested the _access field be an array, so other users and groups can be
specified to have access. There are multiple scenarios in this one alone:

#### 2.1: The Todo List

In this scenario, a user has a reasonable amount of ”personal data” that they want to
selectively share with one or more other users.

#### 2.2: The Chat/Forum/Newsgroup

In this scenario, a user wants to share any number of documents with a reasonable number of
groups. However, since we need to limit the number of groups a user belongs to (currently
10, see below for details), this might actually not be a great solution. Or folks couldn’t
be in more than 10 chat groups at a time.

#### 2.3: The Corporate Hierarchy

In this scenario, users want to share any number of docs with a reasonable number of groups
in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions,
divisions report up to their one executive, etc.

### 3: Multiple Apps

The preceding scenarios all assume that a single application is responsible for everything.
However, once we allow mutually distrusting users into a single database *and* make each per-user
slice work (almost) like a full standalone CouchDB database, what would stop users from using
this for a multi-homing feature, where different applications are used for each user in the
same database?

I’ll be referring to these scenarios down the line.

* * *

## Design Docs

### Admin

One of the downsides of db-per-user is managing design docs in the face of a changing application,
that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible,
but tedious. In all scenarios above but scenario 3., we could simplify this significantly.
Say an admin creates a design doc, and gives all users in the db access to this design doc
(this could be with the _users role, or yet another new role _members, if we need it), requesting
the result of a view defined in that design doc will produce an index that is powered by the
requesting user’s by-access-seq index section(s).

N.B., this would require us to change a fundamental assumption when doing the association
between a design doc’s definition and index: normally, there is only the `views` member
that is hashed and that hash is used as the index’s filename. Because there is only by-seq
to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq,
any view index built will have to take a user’s name and roles into account. When a user
leaves a group, or gains a group, all indexes for that user will no longer be valid and need

### User

In any of the scenarios above, but especially 3., there could be legitimate per-user design
docs, so how should those be treated in an _access enabled database?

The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll
skip over the deprecated _show, _list, and _update).

The easiest to handle is a `filters`: if a user specifies a filter for a _changes request
or replication that lives in a design doc they don’t have access to, they get an error,
similar to if they specify a non-existent design doc, just with `unauthorized` instead of

Next `views` is also not very hard to imagine working: just like globally defined views for
that db, the index is built for each user based on the user’s name and roles.

More troubling are `validate_doc_update` functions: One, they are already troubling in that
they slow down any document updates. Two, if we now import an existing db-per-user scenario
where each user has their own design docs, how should we apply validate_doc_update functions?
10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management
of VDUs that are active on a database. One option would be to ignore VDUs if they are not
defined globally (say with a _members role). But especially in scenario 3. this becomes problematic,
but even without that specific scenario, this violates the no surprises best practice.

We could say:

a) we don’t support scenario 3.
b) we find a complicated but efficient way to apply only those VDUs that are defined in design
docs the writing user has access to plus any global ones (this would be neat but rather complicated
and potentially still impractical from a performance perspective for N users).
c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.

I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate
db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off
between the most common scenarios for db-per-user while keeping the implementation manageable.
Globally accessible design docs would show up in a user’s changes feed and would replicate
down to say a PouchDB application which might be the exclusive user of those design docs.

In practice this would mean, a document that has an _id that starts with _design/ will have
to be produced by a database admin. Luckily, that’s already the case. We should just make
sure that folks don’t give db-admin access to all users habitually.

## Read and Write Access

Speaking of validate_doc_update, it is used for two things: checking document schema and doc
update authorisation.

Once we allow access to a document with an _access field, we need to decide what kind of access
this gives to a doc: read-only or read-write (I’m not considering write-only because for
anything but doc creations this is not useful as you need access to the current _rev).

However, when we look at implementing an application on top of our existing API, it is already
weird that read access can be controlled globally (or with _access on a per doc level), but
write access requires writing JavaScript code. I think it would be a reasonable expectation
for users to expect a per-doc read/write permission granting.

So we could have all of the above, but with two extra fields: _access_read and _access_write,
or _access: {read: [], write: []} or we overload user and group names: _access: [user_a:read,
user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring
characters in group names.

The former seems more explicit, but from an API perspective that’s a little more awkward:
remember that we currently have an arbitrary limit of 10 members in a user’s role array,
to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away
with more, more easily however. If we allow the specification of access control in two lists,
and one of the lists implies membership in the other, we have a total limit of 10 members
across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird,
but doable. Anyway, good bikeshed.

* * * 

So far. I think all of the problems outlined are solvable, if with a clear definition of what
use-cases we do not support with access. If you have more scenarios than the ones I outlined,
please add them and we can see if they cause any additional trouble.

Thanks for reading this far and I’m looking forward to your feedback.

Jan “_access” Lehnardt

> On 17. Feb 2019, at 15:25, Jan Lehnardt <> wrote:
> Hi Everyone,
> I’m happy to share my work in progress attempt to implement the per-doc access control
feature we discussed a good while ago:
> You can check out my branch here:
> <>
> It is very much work in progress, but it is far enough along to warrant discussion.
> The main point of this branch is to show all the places that we would need to change
to support the proposal.
> Things I’ve left for later:
> - currently only the first element in the _access array is used. Our and/or syntax can
be added later.
> - building per-access views has not been implemented yet, couch_index would have to be
taught about the new per-access-id index.
> - pretty HTTP error handling
> - tests except for a tiny shell script 😇
> Implementation notes:
> You create a database with the _access feature turned on like so:  PUT /db?access=true
> I started out with storing _access in the document body, as that would allow for a minimal
change set, however, on doc updates, we try hard not to load the old doc body from the database,
and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended
the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored
in both by-id and by-seq. I will need guidance on how extending these records impact multi-version
cluster interop. And especially whether this is an acceptable approach.
> * * *
> The main addition is a new native query server called couch_access_native_proc, which
implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass
in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those
docs that match the username and roles in their _access property. The existing handlers for
_all_docs and _changes have been augmented to use the new indexes instead of the default ones,
unless the user is an admin.
> * * *
> The rest of the diff is concerned with making document CRUD behave as you’d expect
it. See this little demonstration for what things look like:
> <>
(I’m just noticing that there might be something wonky with DELETE, but you’ll get the
gist #rimshot)
> * * *
> Open questions:
> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing
that is new however which would require all apps to be changed is that for an _access enabled
database to include an _access field in their docs (docs with no _access are admin-only for
now). We might want to consider on new document writes to auto-insert the authenticated user’s
name as the first element in the _access array, so existing apps “just work”.
> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you
have a lot of users, but making those per-user requests inside an _access enabled database
efficient would be doubly nice, so why not use the username from the first question above
and use that as the partition key? This would work nicely for natural users with their own
docs that want to share them with others later, but I can easily imagine a pipelined use of
CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over
and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality
rule of partitions (have a lot of small ones), instead all docs go through all three users.
I’d be okay with treating the later scenario as a minor use-case, but for that use-case,
we should be able to disable auto-partitioning on db creation.
> - building access view indexes for docs that have frequent _access changes, lead to many
orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes
in case folks just swap back and forth).
> * * *
> I’ll leave this here for now, I’m sure there are a few more things to consider.
> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
> Best
> Jan
> —

Professional Support for Apache CouchDB:

View raw message