couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Willem van der Westhuizen <wil...@kwantu.net>
Subject Re: Maximum number of databases?
Date Sun, 09 Feb 2020 16:37:18 GMT
I was wondering if the Cloudant recommendation was based on the Cloudant 
superstructure, or the underlying couchdb architecture. And particularly 
how important the issue of continuous replication on each of those is in 
the assessment. Here is our use case:

Each user has their own user database, which is mirrored on the local 
pouchdb client. (in the browser, electron offline, apk offline). We have 
an "online" mode, in which data objects are read directly from couchdb 
(shared database - we do not use per user databases for access control, 
but for improved performance over poor networks). It saves any document 
in a local db cache for working. In our use case, which is a business 
process management reporting tool, there are always a number of 
documents in the packet to be processed. It is important that all the 
documents save correctly, or none at all. Therefore, when the user does 
the final submit, all the documents are processed to the user's local 
copy of the User database, not the shared one. From here the list of 
documents are packaged into a transaction object (which can be quite 
large) and replicated to the users' local copy on the server. From here, 
the transaction manager picks up the new document, processes it and 
saves it back into the shared database as part of the transaction process.

Because we use a one-way packet driven replication, triggered by a save 
event and not a continuous replication, this limits the performance 
issues (so we believe) as long as the transaction manager can process 
all the incoming documents effectively. And that can be scaled up 
without too much difficulty.

I would be interested to hear if there is a reason that we should be 
concerned?

Willem

On 2020/02/09 18:02, Marcus wrote:
> How many databases can be used without causing issues with replication and server performance?
>
> I found two very different opinions. The pouchdb blog quotes 100K (based on a discussion
about Cloudant in 2014). However a Cloudant blog series from March 2019 recommends a maximum
of 500.
>
> Can anyone explain the huge difference? I understand it's going to depend on use cases,
but a difference of 90,500 databases is significant.
>
> 500 are too few when databases are needed for read access control using roles. One for
each user's personal document locker, one for public data (web), and one for a private group.
That leaves about 160 users.
>
> Here are two excerpts from that Cloudant blog series of March 2019.
>
> "Rule 4: Fewer databases are better than many
>
> If you can, limit the number of databases per Cloudant account to 500 or fewer. While
there is nothing magical about this particular number (Cloudant can safely handle more), there
are several use cases that are adversely affected by large numbers of databases in an account."
>
> "Rule 5: Avoid the “database per user” anti-pattern like the plague
> If you’re building out a multi-user service on top of Cloudant, it is tempting to let
each user store their data in a separate database under the application account. That works
well, mostly, if the number of users is small."
>
> Source: https://www.ibm.com/cloud/blog/cloudant-best-and-worst-practices-part-1
>
> What are your personal experiences with large numbers of databases?
>
> Marcus
>
>

Mime
View raw message