couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Vander Wilt <nate-li...@calftrail.com>
Subject Re: Half-baked idea: incremental virtual databases
Date Wed, 30 Jan 2013 01:53:33 GMT
So I've heard from both hosting providers that it is fine, but also managed to take both of
their shared services "down" with only about ~100 users (200 continuous filtered replications).
I'm only now at the point where I have tooling to build out arbitrary large tests on my local
machine to see the stats for myself, but as I understand it the issue is that every replication
needs at least one couchjs process to do its filtering for it.

So rather than inactive users mostly just taking up disk space, they're instead costing a
full-fledged process worth of memory and system resources, each, all the time. As I understand
it, this isn't much better on BigCouch either since the data is scattered ± evenly on each
machine, so while the *computation* is spread, each node in the cluster still needs k*numberOfUsers
couchjs processes running. So it's "scalable" in the sense that traditional databases are
scalable: vertically, by buying machines with more and more memory.

Again, I am still working on getting a better feel for the costs involved, but the basic theory
with a master-to-N hub is not a great start: every change needs to be processed by every N
replications. So if a user writes a document that ends up in the master database, every other
user's filter function needs to process that change coming out of master. Even when N users
are generating 0 (instead of M) changes, it's not doing M*N work but there's still always
2*N open connections and supporting processes providing a nasty baseline for large values
of N.

After another day considering my options, I see two (complementary) ways to mitigate this
more simply:
1. Have developers expect the need to scale out via intermediary databases with intermediary
filters, arrangement tailored manually for the application at hand. This burns more storage
space, but is consistent with CouchDB's "disks are cheap" philosophy. Assuming it's possible
to combine replication filters for a given application, this should make it more like M*log(N)
cost.
2. Implement a lesser (more "eventual") form of "continuous" replication — replications
request a priority and let CouchDB "continuously" replicate without actually holding every
database open (catch up every few seconds/minutes/hours  as available resources permit). This
would lower the baseline resources required.

Both of these currently require extra tooling and infrastructure not easily handled from a
design document. I didn't expect to need a full set of maintenance scripts and sharding strategies
at a hundred not-very-active users — since continuous filtered replication is a fundamental
pattern for communicating databases, it'd be nice if it were scalable, or at least cheaper.

thanks,
-natevw



On Jan 29, 2013, at 4:55 PM, Stephen Bartell wrote:

> Nathan, I'm actually in the process of setting up a multi-tenant environment  the canonical
way, like you have.
> 
> I've seen the replication overhead get pretty intense, but I figure that scaling out
to several couches is the way to go once the overhead becomes unbearable.  Actually I was
hoping BigCouch would eventually be the answer.  
> 
> Why is this not the case for you? 
> 
> In one of those links you provided (JasonSmith@stackoverflow) said that db per user is
the only scalable way.  It would be nice if he or someone here could weight in on why/how
thats the only scalable way. Especially in light of Nathan claiming the exact opposite.
> 
> sb
> 
> On Jan 29, 2013, at 10:44 AM, Nathan Vander Wilt <nate-lists@calftrail.com> wrote:
> 
>> # The problem
>> 
>> It's a fairly common "complaint" that CouchDB's database model does not support fine-grained
control over reads. The canonical solution is a database per user:
>> http://wiki.apache.org/couchdb/PerDocumentAuthorization#Database_per_user
>> http://stackoverflow.com/a/4731514/179583
>> 
>> This does not scale.
>> 
>> 1. It complicates formerly simple backup/redundancy: now I need to make sure N replications
stay working, N databases have correct permissions, instead of just one "main" database. Okay,
write some scripts, deploy some cronjobs, can be made to work...
>> 
>> 2. ...however, if data needs to be shared between users, this model *completely falls
apart*. Bi-directional continuous filtered replication between a "hub" and each user database
is extremely resource intensive.
>> 
>> I naïvely followed the Best Practices and ended up with a system that can barely
support 100 users to a machine due to replication overhead. Now if I want to continue doing
it "The Right Way" I need to cobble together some sort of rolling replication hack at best.
>> 
>> It's apparent the real answer for CouchDB security, right now, is to hide the database
underneath some middleware boilerplate crap running as DB root. This is a well-explored pattern,
by which I mean the database ends up with as many entry points as a sewer system has grates.
>> 
>> 
>> # An improvement?
>> 
>> What if CouchDB let you define virtual databases, that shared the underlying document
data when possible, that updated incrementally (when queried) rather than continuously, that
could even internally be implemented in a fanout fashion?
>> 
>> - virtual databases would basically be part of the internal b-tree key hierarchy,
sort of like multiple root nodes sharing the branches as much as possible
>> - sharing the underlying document data would almost halve the amount of disk needed
versus a "master" database storing all the data which is then copied to each user
>> - updating incrementally would put less continuous memory pressure on the system
>> - haven't actually done the maths, so I may be missing something, but wouldn't fanning
out changes internally from a master database through intermediate partitions reduce the processing
load?
>> 
>> Basically, rather than each time a user updates a document, copying it to a master
database, then filtering every M updates through N instances of couchjs; instead internally
CouchDB could build a tree of combined filters — say, master database filters to log(N)
hidden partitions at the first level and accepted changes would trickle through only relevant
further layers. (In a way, this is kind of at odds with the incremental nature — maybe it
does make sense to pay an amortized cost on write rather than on reads.)
>> 
>> 
>> # The urgency
>> 
>> Maybe this *particular* solution isn't really a solution, but we need one:
>> 
>> If replicating amongst per-user databases is the only correct way to implement document-level
read permissions, CouchDB **NEEDS** built-in support for a scalable way of doing so.
>> 
>> There are plenty of other feature requests I could troll the list with regarding
CouchApps. But this one is key; everything else I've been able to work around behind a little
reverse proxy here and in front of an external process there. Without scalable read-level
security, I see no particular raison d'être for Apache CouchDB — if CouchDB can't support
direct HTTP access in production in general, then it's just another centralized database.
>> 
>> 
>> thanks,
>> -natevw
> 


Mime
View raw message