From user-return-5693-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Wed Jul 29 09:28:52 2009 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 63291 invoked from network); 29 Jul 2009 09:28:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 29 Jul 2009 09:28:52 -0000 Received: (qmail 32464 invoked by uid 500); 29 Jul 2009 09:28:51 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 32373 invoked by uid 500); 29 Jul 2009 09:28:51 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 32363 invoked by uid 99); 29 Jul 2009 09:28:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2009 09:28:51 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of b.candler@pobox.com designates 208.72.237.25 as permitted sender) Received: from [208.72.237.25] (HELO sasl.smtp.pobox.com) (208.72.237.25) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2009 09:28:40 +0000 Received: from localhost.localdomain (unknown [127.0.0.1]) by a-sasl-quonix.sasl.smtp.pobox.com (Postfix) with ESMTP id E3DC918E86; Wed, 29 Jul 2009 05:28:17 -0400 (EDT) Received: from mappit (unknown [80.45.95.114]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by a-sasl-quonix.sasl.smtp.pobox.com (Postfix) with ESMTPSA id 7C1FB18E85; Wed, 29 Jul 2009 05:28:11 -0400 (EDT) Received: from brian by mappit with local (Exim 4.69) (envelope-from ) id 1MW5SD-0002x7-PU; Wed, 29 Jul 2009 10:28:09 +0100 Date: Wed, 29 Jul 2009 10:28:09 +0100 From: Brian Candler To: Robert Ames Cc: user@couchdb.apache.org Subject: Re: Re: How reliable is the versioning system? Message-ID: <20090729092809.GA10984@uk.tiscali.com> References: <674809.8364.qm@web31807.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <674809.8364.qm@web31807.mail.mud.yahoo.com> User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-Pobox-Relay-ID: 2631BE3E-7C22-11DE-BC40-F699A5B33865-28021239!a-sasl-quonix.pobox.com X-Virus-Checked: Checked by ClamAV on apache.org On Sun, Jul 26, 2009 at 02:19:23PM -0700, Robert Ames wrote: In addition, I > see here an explicit recommendation is to maintain revision history > outside of CouchDB, and it seems as though the replication model is pretty > similar to Git's model... The replication model is quite different to Git's - although this would be a useful comparison to put on the wiki somewhere. With Git, you have a copy of each peer's tree (remotes/PEER/BRANCH). Then you perform a merge into a single document in your working tree. If the merge fails, then it fails; you still have a single working copy, but with the conflicts explicitly marked within that document. It's up to you to resolve those conflicts and commit the final version. With Git, merging is always done when you *pull* from a peer; if you *push* to a peer which can't be fast-forwarded, the push fails. (Or you can force the push, but that simply overwrites the changes at the peer) With CouchDB, you don't keep track of peers in the database. When you replicate from a peer, or a peer replicates to you, and the documents conflict(*), then you get multiple copies of the document within the database. When you request a document by ID, you get an arbitrary one of this set, unless you explicitly ask for the other versions. However the multiple copies are all there and are all effectively equal copies (except for the property that one is arbitrary chosen as the "winner"); no precedence is given to the version which originated locally, for instance. (*) In this case, 'conflict' effectively means 'derived from a different _rev'. There is no attempt in CouchDB to perform any merging. A CouchDB-like system running on top of git would be extremely interesting. I can see four main parts: - a high-performance git backend which appends objects to a single file. The compact operation creates a new file and rotates it into place, and ideally retains git's ability to create packs and compact using diffs. - a new 'btree' git object class, for mapping an unlimited number of keys to objects. refs would be stored using this. The existing flat 'tree' object won't scale well to millions of keys. - a HTTP interface for storing and retrieving objects by key, using the 'btree' class again (*) - the map/reduce engine ported to run on top of this, following the commit tree to determine which documents had changed. There would need to be some way to handle merging and conflicts. Perhaps the HTTP interface returns all peer versions as a multipart, with their commit IDs as the rev. It also needs to be decided how to handle 'attachments'. Personally I'd like to store the MIME type against every document, which means you could store non-JSON objects as first-class objects in their own right. Then you could do things like using map/reduce to scale JPEGs. (*) Note: you might think of having one branch per document, instead of a top-level btree object containing all documents. That would give each object an independent history. The trouble with that approach is that git replication doesn't work well with thousands of branches - I've tried it :-) - because it has to iterate linearly over each branch. So I think that you really need one btree object for the database, and then store the history of that via commits. Regards, Brian.