couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <m...@jan.io>
Subject Re: [DISCUSS] Per-doc access control
Date Sun, 10 Mar 2019 15:57:48 GMT
One addition, the slotting in of _access into existing security mechanisms is as follows:

1. Check if a user is in _security
2. If yes, check it user is in _access (modulo read/write)
3. If yes, does the doc update pass any globally defined VDUs
4. If yes, operation can proceed.

Cheers
Jan
—

> On 10. Mar 2019, at 15:51, Jan Lehnardt <jan@apache.org> wrote:
> 
> Hey all,
> 
> after mulling this over some more, I’d like to tackle the detailed API and behaviour
for this. Especially how _access work in conjunction with existing access control features.
> 
> My guiding principles so far are:
> 
> 1. Make the API intuitive, things should work like they look like they should work like.
> 2. The default should never be that a resources is accidentally left accessible to the
public.
> 3. This should work as a natural extension to the existing security features*.
> 
> * I’d be up for reworking the whole lot, too, but that might be a better discussion
for > 4.0.
> 
> 
> ## Database Creation and Default Behaviours
> 
> Creating a database with _access features is, as mentioned before done via a flag to
PUT /database?access=true
> 
> In a 3.0 world where this would land, we already agreed that databases should be admin-only
by default (instead of world read/writeable today). This is a sensible default, but that leaves
us with an _access enabled database that can’t be used by anyone by server or db admins.
Not very useful.
> 
> To allow arbitrary users to use the db, I suggest we use the existing _security system:
i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside
of _security, they can proceed and create documents on the db. This puts a second step burden
on the application developer, but it slots cleanly into the existing security mechanisms,
and doesn’t require special case handling. Alternatively, we could define that _security
isn’t available in _access enabled databases, but that’s something I’d like to avoid
if at all possible.
> 
> In order to make it easy to specify that “everyone in _users” should be able to use
the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone
in /_users” (this only excludes server admins which have full access anyway).
> 
> * * *
> 
> 
> ## Document Creation and Access Control
> 
> Next, one of our non-admin users creates a doc. There are multiple options as to how
we store the _access information.
> 
> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the
first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}.
This is a little bit counter-intuitive.
> 
> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit
granting of access permissions on doc creation and I think is preferable.
> 
> This leaves the edge case of docs that have no _access member: so far I thought those
docs are admin-only, with maybe a db-wide option to swap the default to public access, but
I think given the explicitness of 2. we can do better: require _access for all new doc creations
in access-enabled databases. A user can not create a new document without an _access field
that is an array that has at least one member. For public documents, we could invent a new
role _public, and admin-only docs could use the existing role _admin.
> 
> The one downside to this approach is that we won’t be able to replicate existing databases
into an access-enabled database without modifying all documents. This might be a worthwhile
trade-off, but we should make that decision consciously and document it well. We could allow
for a special case where an _admin user can create docs that have no _access field, and those
docs are treated as having only the _admin role in _access. So at least we could replicate
all data in, but then require a manual step to update all docs to say, migrate an existing
db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
> 
> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name
or role in its _access field.
> 
> For both creations and updates, a user MUST supply at least one role they belong to or
their own username.
> 
> * * *
> 
> 
> ## _revs_diff
> 
> /db/_revs_diff can answer the question of which revisions of a document do NOT exist
on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
> 
> This would allow users to specify ids and rev(s) for docs they don’t have access too
(anymore), so the result schema should be expanded to handle id: unauthorized or somesuch,
something the replicator needs to know what to do with, if it encounters it (say a user got
removed from the _access list inbetween the replicator opening _changes and requesting the
doc).
> 
> The _revs_diff implementation would have to altered to send an unauthorized token for
each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes,
or any other performance optimisation, that’d be great. I haven’t looked at that code
at all, yet.
> 
> An important side-effect of this is, once a user has been added to a doc’s _access
list, they get access to “the full history of the doc”, even before they had access. Of
course, in CouchDB this means only getting access to the rev ids, and not the content, but
since they are content-addressable hashes, a user could brute-force themselves into revealing
certain real values from earlier incarnations of the doc. I’d rather not track _access per
document revision in perpetuity, so this is something we have to be very up-front about.
> 
> * * *
> 
> 
> ## Partitioned Databases
> 
> I mentioned partitioned databases in my previous mail, and I think it is something we
can document that end-users can opt into, but doesn’t require any special casing on the
_access proposal. That is, if users start prefixing their doc ids with a user name or id and
enable both _access and partitions, then they get all the benefits of a partitioned database,
and if they choose not to, they don’t, but things keep working. There are enough use-cases
to warrant both behaviours.
> 
> * * *
> 
> 
> ## Scenarios that _access should help with.
> 
> Overall, we developed _access to allow users to stop using the db-per-user architecture,
but once we have per-doc-access control, folks might start using this for all manner of things.
We should be clear about which scenarios we support and which we don’t.
> 
> 
> ### Scenario 1: db-per-user
> 
> In this scenario, _access enabled databases, the only way to allow mutually untrusting
users to store data in a part of CouchDB that only they (and admins) have access to was giving
each user their own database.
> 
> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own
docs knowing no other user (aside from admins) can access those docs.
> 
> This is the simplest scenario, as all we’d have to track the owner of a document and
produce by-access-id/seq indexes based on that owner.
> 
> The current prototype implementation mostly reflects this stage. Not saying this is what
we should ship, but it is the easiest do implement and explain.
> 
> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks
who don’t need anything else.
> 
> 
> ### Scenario 2: db-per-user + Sharing
> 
> The second we allow per doc auth, users will want to share those docs with other users.
That’s why we initially suggested the _access field be an array, so other users and groups
can be specified to have access. There are multiple scenarios in this one alone:
> 
> #### 2.1: The Todo List
> 
> In this scenario, a user has a reasonable amount of ”personal data” that they want
to selectively share with one or more other users.
> 
> #### 2.2: The Chat/Forum/Newsgroup
> 
> In this scenario, a user wants to share any number of documents with a reasonable number
of groups. However, since we need to limit the number of groups a user belongs to (currently
10, see below for details), this might actually not be a great solution. Or folks couldn’t
be in more than 10 chat groups at a time.
> 
> #### 2.3: The Corporate Hierarchy
> 
> In this scenario, users want to share any number of docs with a reasonable number of
groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with
divisions, divisions report up to their one executive, etc.
> 
> 
> ### 3: Multiple Apps
> 
> The preceding scenarios all assume that a single application is responsible for everything.
However, once we allow mutually distrusting users into a single database *and* make each per-user
slice work (almost) like a full standalone CouchDB database, what would stop users from using
this for a multi-homing feature, where different applications are used for each user in the
same database?
> 
> I’ll be referring to these scenarios down the line.
> 
> * * *
> 
> 
> ## Design Docs
> 
> ### Admin
> 
> One of the downsides of db-per-user is managing design docs in the face of a changing
application, that is, how to distribute new design docs across 10s of 1000+s of user dbs?
It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify
this significantly. Say an admin creates a design doc, and gives all users in the db access
to this design doc (this could be with the _users role, or yet another new role _members,
if we need it), requesting the result of a view defined in that design doc will produce an
index that is powered by the requesting user’s by-access-seq index section(s).
> 
> N.B., this would require us to change a fundamental assumption when doing the association
between a design doc’s definition and index: normally, there is only the `views` member
that is hashed and that hash is used as the index’s filename. Because there is only by-seq
to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq,
any view index built will have to take a user’s name and roles into account. When a user
leaves a group, or gains a group, all indexes for that user will no longer be valid and need
rebuilding.
> 
> 
> ### User
> 
> In any of the scenarios above, but especially 3., there could be legitimate per-user
design docs, so how should those be treated in an _access enabled database?
> 
> The significant fields in a design doc are `views`, `validate_doc_update` and `filters`
(I’ll skip over the deprecated _show, _list, and _update).
> 
> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request
or replication that lives in a design doc they don’t have access to, they get an error,
similar to if they specify a non-existent design doc, just with `unauthorized` instead of
`not_found`.
> 
> Next `views` is also not very hard to imagine working: just like globally defined views
for that db, the index is built for each user based on the user’s name and roles.
> 
> More troubling are `validate_doc_update` functions: One, they are already troubling in
that they slow down any document updates. Two, if we now import an existing db-per-user scenario
where each user has their own design docs, how should we apply validate_doc_update functions?
10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management
of VDUs that are active on a database. One option would be to ignore VDUs if they are not
defined globally (say with a _members role). But especially in scenario 3. this becomes problematic,
but even without that specific scenario, this violates the no surprises best practice.
> 
> We could say:
> 
> a) we don’t support scenario 3.
> b) we find a complicated but efficient way to apply only those VDUs that are defined
in design docs the writing user has access to plus any global ones (this would be neat but
rather complicated and potentially still impractical from a performance perspective for N
users).
> c) we could store all per-user design docs, but ignore them completely, VDUs, views and
filters.
> 
> I think I currently fall on the side of not supporting scenario 3. and asking folks who
migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a
good trade-off between the most common scenarios for db-per-user while keeping the implementation
manageable. Globally accessible design docs would show up in a user’s changes feed and would
replicate down to say a PouchDB application which might be the exclusive user of those design
docs.
> 
> In practice this would mean, a document that has an _id that starts with _design/ will
have to be produced by a database admin. Luckily, that’s already the case. We should just
make sure that folks don’t give db-admin access to all users habitually.
> 
> 
> ## Read and Write Access
> 
> Speaking of validate_doc_update, it is used for two things: checking document schema
and doc update authorisation.
> 
> Once we allow access to a document with an _access field, we need to decide what kind
of access this gives to a doc: read-only or read-write (I’m not considering write-only because
for anything but doc creations this is not useful as you need access to the current _rev).
> 
> However, when we look at implementing an application on top of our existing API, it is
already weird that read access can be controlled globally (or with _access on a per doc level),
but write access requires writing JavaScript code. I think it would be a reasonable expectation
for users to expect a per-doc read/write permission granting.
> 
> So we could have all of the above, but with two extra fields: _access_read and _access_write,
or _access: {read: [], write: []} or we overload user and group names: _access: [user_a:read,
user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring
characters in group names.
> 
> The former seems more explicit, but from an API perspective that’s a little more awkward:
remember that we currently have an arbitrary limit of 10 members in a user’s role array,
to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away
with more, more easily however. If we allow the specification of access control in two lists,
and one of the lists implies membership in the other, we have a total limit of 10 members
across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird,
but doable. Anyway, good bikeshed.
> 
> 
> * * * 
> 
> 
> So far. I think all of the problems outlined are solvable, if with a clear definition
of what use-cases we do not support with access. If you have more scenarios than the ones
I outlined, please add them and we can see if they cause any additional trouble.
> 
> Thanks for reading this far and I’m looking forward to your feedback.
> 
> 
> Best,
> Jan “_access” Lehnardt
> —
> 
> 
> 
> 
>> On 17. Feb 2019, at 15:25, Jan Lehnardt <jan@apache.org> wrote:
>> 
>> Hi Everyone,
>> 
>> I’m happy to share my work in progress attempt to implement the per-doc access
control feature we discussed a good while ago:
>> 
>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
<https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>> 
>> You can check out my branch here:
>> 
>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>> 
>> It is very much work in progress, but it is far enough along to warrant discussion.
>> 
>> The main point of this branch is to show all the places that we would need to change
to support the proposal.
>> 
>> Things I’ve left for later:
>> 
>> - currently only the first element in the _access array is used. Our and/or syntax
can be added later.
>> - building per-access views has not been implemented yet, couch_index would have
to be taught about the new per-access-id index.
>> - pretty HTTP error handling
>> - tests except for a tiny shell script 😇
>> 
>> Implementation notes:
>> 
>> You create a database with the _access feature turned on like so:  PUT /db?access=true
>> 
>> I started out with storing _access in the document body, as that would allow for
a minimal change set, however, on doc updates, we try hard not to load the old doc body from
the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive,
so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute
that is stored in both by-id and by-seq. I will need guidance on how extending these records
impact multi-version cluster interop. And especially whether this is an acceptable approach.
>> 
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>> 
>> * * *
>> 
>> The main addition is a new native query server called couch_access_native_proc, which
implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass
in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those
docs that match the username and roles in their _access property. The existing handlers for
_all_docs and _changes have been augmented to use the new indexes instead of the default ones,
unless the user is an admin.
>> 
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>> 
>> * * *
>> 
>> The rest of the diff is concerned with making document CRUD behave as you’d expect
it. See this little demonstration for what things look like:
>> 
>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497>
(I’m just noticing that there might be something wonky with DELETE, but you’ll get the
gist #rimshot)
>> 
>> * * *
>> 
>> Open questions:
>> 
>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One
thing that is new however which would require all apps to be changed is that for an _access
enabled database to include an _access field in their docs (docs with no _access are admin-only
for now). We might want to consider on new document writes to auto-insert the authenticated
user’s name as the first element in the _access array, so existing apps “just work”.
>> 
>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if
you have a lot of users, but making those per-user requests inside an _access enabled database
efficient would be doubly nice, so why not use the username from the first question above
and use that as the partition key? This would work nicely for natural users with their own
docs that want to share them with others later, but I can easily imagine a pipelined use of
CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over
and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality
rule of partitions (have a lot of small ones), instead all docs go through all three users.
I’d be okay with treating the later scenario as a minor use-case, but for that use-case,
we should be able to disable auto-partitioning on db creation.
>> 
>> - building access view indexes for docs that have frequent _access changes, lead
to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep
1-N indexes in case folks just swap back and forth).
>> 
>> * * *
>> 
>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>> 
>> I’d love to hear any and all feedback you might have. Especially if anything is
unclear.
>> 
>> Best
>> Jan
>> —
> 
> -- 
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
> 


Mime
View raw message