incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Scaling db-per-user (Was: Re: Disabling doc include)
Date Thu, 02 Jan 2014 20:17:27 GMT

On 02 Jan 2014, at 21:08 , Jens Alfke <jens@couchbase.com> wrote:

> 
> On Jan 2, 2014, at 11:56 AM, Jan Lehnardt <jan@apache.org> wrote:
> 
>> Out of curiosity, what scaling limit have you found? Is this documented somewhere?
> 
> By “we found” I should have said “we extrapolated”. We have customers that will
need hundreds of thousands of user accounts, and attaching that many replications to a central
master database wouldn’t be practical. Especially since they’d be filtered replications
— every time a document was added to the master database, CouchDB would have to run n filter
functions to decide whether to push it to n different user databases. Another scaling factor
is that, if documents are accessible to many users, the storage space needed for those documents
will be greatly multiplied since many replicas of them will exist.
> 
> If someone only needs a few hundred user or user-subset databases, though, this should
be a feasible approach.

Thanks Jens. Sounds sensible to me.

We added /_db_updates in 1.4.0 that allows building the above with the difference that a replication
only runs for active users, thus delaying most of the work until it is needed *and* avoiding
having to run hundreds of thousands of replications at the same time. Managing all that however
is still an exercise left to the user (I know of two implementations of this in node) and
we should see if we can smart up the replicator accordingly.

Storage aside, do you thing that would worked for your scenario?

Mime
View raw message