couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Anderson <jch...@apache.org>
Subject Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Date Fri, 20 Feb 2009 01:40:20 GMT
On Thu, Feb 19, 2009 at 5:34 PM, Shaun Lindsay <shaun@meebo.com> wrote:
> Hi all,
> So, a couple months ago we implemented almost exactly the couch
> clustering/partitioning solution described below.  The couch cluster (which
> we called 'The Lounge') sits behind nginx running a custom module that farms
> out the GETs and PUTs to the appropriate node/shard and the views to a
> python proxy daemon which handles reducing the view results from the
> individual shards and returning the full view.  We have replication working
> between the cluster nodes so the shards exist multiple places and, in the
> case of one of the nodes going down, the various proxies fail over to the
> backup shards.
>
> This clustering setup has been running in full production for several months
> now with minimal problems.
>
> We're looking to release all the code back to the community, but we need to
> clear it with our legal team first to make sure we're not compromising any
> of our more business-specific, proprietary code.
>
> In total, we have:
> a nginx module specifically set up for sharding databases
> a 'smartproxy', written in Python/Twisted, for sharding views
> and a few other ancillary pieces (replication notification, view updating,
> etc)
>
> Mainly, I just wanted to keep people from duplicating the work we've done --
> hopefully we can release something back to the community in the next several
> weeks.
>
> We're having a meeting tomorrow morning to figure out what we can release
> right now (probably the nginx module, at the least).  I'll let everyone know
> what out timeline looks like.
>

This is really cool. It sounds like a tool people could get started
with right away to build big CouchDB clusters. Is there anything you
can tell us about the size of clusters you've used?

The smartproxy code will probably be a really good illustration of
what we'll probably want to implement in Erlang for CouchDB,
eventually. Again, +1 for software people can use today.

Hopefully it turns out to be easy to release!

Cheers,
Chris

-- 
Chris Anderson
http://jchris.mfdz.com

Mime
View raw message