On 20 Feb 2009, at 02:34, Shaun Lindsay wrote: > Hi all, > So, a couple months ago we implemented almost exactly the couch > clustering/partitioning solution described below. Shaun, this sounds fantastic! :) I hope you can release the code for this. Cheers Jan -- > The couch cluster (which > we called 'The Lounge') sits behind nginx running a custom module > that farms > out the GETs and PUTs to the appropriate node/shard and the views to a > python proxy daemon which handles reducing the view results from the > individual shards and returning the full view. We have replication > working > between the cluster nodes so the shards exist multiple places and, > in the > case of one of the nodes going down, the various proxies fail over > to the > backup shards. > > This clustering setup has been running in full production for > several months > now with minimal problems. > > We're looking to release all the code back to the community, but we > need to > clear it with our legal team first to make sure we're not > compromising any > of our more business-specific, proprietary code. > > In total, we have: > a nginx module specifically set up for sharding databases > a 'smartproxy', written in Python/Twisted, for sharding views > and a few other ancillary pieces (replication notification, view > updating, > etc) > > Mainly, I just wanted to keep people from duplicating the work we've > done -- > hopefully we can release something back to the community in the next > several > weeks. > > We're having a meeting tomorrow morning to figure out what we can > release > right now (probably the nginx module, at the least). I'll let > everyone know > what out timeline looks like. > > --Shaun Lindsay > Meebo.com > > On Thu, Feb 19, 2009 at 4:48 PM, Chris Anderson > wrote: > >> On Thu, Feb 19, 2009 at 4:35 PM, Ben Browning >> wrote: >>> So, I started thinking about partitioning with CouchDB and realized >>> that since views are just map/reduce, we can do some magic that's >>> harder if not impossible with other database systems. The idea in a >>> nutshell is to create a proxy that sits in front of multiple servers >>> and "sprays" the view queries to all servers, merging the results - >>> hence CouchSpray. This would give us storage and processing >>> scalability and could, with some extra logic, provide data >>> redundancy >>> and failover. >> >> There are plans in CouchDB's future to take care of data >> partitioning, >> as well as querying views from a cluster. Theoretically, it should be >> pretty simple. There are a few small projects that have started down >> the road of writing code in this area. >> >> https://code.launchpad.net/~dreid/sectional/trunk >> >> Sectional is an Erlang http proxy that implements consistent hashing >> for docs. I'm not sure how it handles view queries. >> >> There's also a project to provide partitioning around the basic >> key/value PUT and GET store using Nginx: >> >> http://github.com/dysinger/nginx/tree/nginx_upstream_hash >> >> If you're interested in digging into this stuff, please join dev@. We >> plan to include clustering in CouchDB, so if you're interested in >> implementing it, we could use your help. >> >> Chris >> >> -- >> Chris Anderson >> http://jchris.mfdz.com >>