From couchdb-user-return-214-apmail-incubator-couchdb-user-archive=incubator.apache.org@incubator.apache.org Sun Apr 13 21:25:01 2008 Return-Path: Delivered-To: apmail-incubator-couchdb-user-archive@locus.apache.org Received: (qmail 88806 invoked from network); 13 Apr 2008 21:25:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Apr 2008 21:25:01 -0000 Received: (qmail 74081 invoked by uid 500); 13 Apr 2008 21:25:01 -0000 Delivered-To: apmail-incubator-couchdb-user-archive@incubator.apache.org Received: (qmail 74057 invoked by uid 500); 13 Apr 2008 21:25:01 -0000 Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: couchdb-user@incubator.apache.org Delivered-To: mailing list couchdb-user@incubator.apache.org Received: (qmail 74048 invoked by uid 99); 13 Apr 2008 21:25:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Apr 2008 14:25:01 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.97.50.139] (HELO jan.prima.de) (83.97.50.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Apr 2008 21:24:06 +0000 Received: from [192.168.1.33] (e179076081.adsl.alicedsl.de [::ffff:85.179.76.81]) (AUTH: LOGIN jan, SSL: TLSv1/SSLv3,128bits,AES128-SHA) by jan.prima.de with esmtp; Sun, 13 Apr 2008 21:24:20 +0000 Message-Id: From: Jan Lehnardt To: couchdb-user@incubator.apache.org In-Reply-To: <41fe564f0804121733l15dd9babp64e80b6cd378e23@mail.gmail.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Subject: Re: Relying on revisions for rollbacks Date: Sun, 13 Apr 2008 23:23:46 +0200 References: <41fe564f0804111936t34837fcfv9c71efeb3ffab6ef@mail.gmail.com> <3DC0E48E-88DE-4913-B1BA-518F0CCC4C62@apache.org> <41fe564f0804121733l15dd9babp64e80b6cd378e23@mail.gmail.com> X-Mailer: Apple Mail (2.919.2) X-Virus-Checked: Checked by ClamAV on apache.org Heya Ralf, On Apr 13, 2008, at 02:33, Ralf Nieuwenhuijsen wrote: >> >> Because the storage system is pretty wasteful and you'd end up with >> several Gigabytes of database files for just a few hundred >> Megabytes of >> actual data. So we do need compaction in one form or another. A >> compaction >> that retains revisions is a lot harder to write. Also, dealing with >> revisions in a distributed setup is less than trivial and would >> complicate >> the replication system quite a bit. > > > The gigabytes versus hundred megabytes seem acceptable to me. > Esspecially > when we can scale that easily. Also, it seems to depend on how often > data > changes. A simple solution to compact revisions would be to store each > revision as a reverse-diff as well. The normal data can then be > compacted, > whereas the reverse-diff is just kept. From the most recent version > the > older versions can be established. > > Question 1: How would manual revisions be any more space efficient? Manual revisions as top-level documents could be compacted to under 2N of the data size not 100N, where N is the size of your actual data. Of course, the amount of data you want to store won't magically decrease. It is rather that CouchDB's storage engine trades disk-space for speed and consistency at runtime with asynchronous compaction to regain wasted space. And, just to make sure, quoting myself: "The revisions are not, at least at this point, meant to implement revision control systems, they rather exists for the optimistic concurrency control that allows any number of parallel readers while serialised writes are happening and to power replication." >> Compacting is a manual process at the moment. If we would introduce a >> scheduling mechanism, it would certainly be more general purpose >> and you >> could hook in al sorts of operations, including compaction. > > Question 2: In which case 'compacting' (aka as destroying the > revisions) > would still be optional; something we can turn off? You need to run it explicitly at the moment. So by default, everything is kept. This might change in the future, but you will be able to disable it on a database-level. > Question 3: Can we use older revisions in views? No. >> See http://damienkatz.net/2008/02/incremental_map.html >> and http://damienkatz.net/2008/02/incremental_map_1.html >> > Question 4: It appears from the comments this will behave much like a > combinator. So the algorithm complexity of adding one new document > would be > O(1) ? I think so, but I am not the definitive source here. Damien? >> You don't merge, at least at the moment, but declare one revision >> to be the >> winner when resolving the conflict. Since this is a manual process, >> you can >> make sure you don't lose revision trees. Merge might be in at some >> point, >> but no thoughts (at least public) went into that. > > Question 5: Is manually implementing a conflict resolver possible at > the > moment (didn't find it on the wiki) and if so, why not let that > function > just return the winning _data_. That way we could easily implement a > merger. > (which would be a much more sane approach for most documents) It is not possible at the moment. You need to resolve a conflict from your application. There you can do all the merging you need :) >> I don't understand what you mean here :) What is 'doc-is' in this >> context? > > Oops, i meant 'doc-ID's' .. if i have several revisions of the same > document > as seperate documents, then the doc-id can no longer be some nice > name. > Since doc-id's have to be unqiue. Correct. Pretty names can get hairy in a distributed setup, so you might want to stick to UUIDs and provide your own "pretty-name". You wouldn't be able to use that, programatically, except in views, though. >>> The alternative of a cron-like system, could work much like the >>> view-documents. These documents could contain a source url (possibly >>> local), >>> a schedule-parameter and a function that maps a document to an >>> array of >>> documents that is treated as a batch-put. This way we could easily >>> setup >>> replication, but also all kinds of delayed and/or scheduled >>> proccessing >>> of data. >> >> Indeed. No planning went into such a thing at the moment. You might >> want >> to open a feature request at https://issues.apache.org/jira/browse/COUCHDB >> or >> come up with a patch. > > Perhaps i will look into it myself, if it turns out I need this > desperately. > I don't have any erlang experience, but i think my experience with > haskell > will pull me through ;-) Awesome! May recommend Joe Armstrong's 'Programming Erlang' book at http://www.pragprog.com/titles/jaerlang in case you are looking for literature. Thanks for considering helping out. Contributions are very important for the project. >> Conflict resolution and merge functions do sound interesting, I don't >> understand the "not guaranteeing scalability" remark though. In the >> current >> implementation, this feature actually makes CouchDB scalable by >> ensuring, >> that all node participating in a cluster eventually end up with the >> same >> data. If you really do need two-phase-commit (if I understand >> correctly, you >> want that), that would need to be part of your application or a >> intermediate >> storage layer. > > > No, no need for two-phase-commits. Rather, i would suggest the > complete > other extreme. No failed inserts/updates ever, including batch puts. > Just a > generic merging conflict solver. > > JSON seems very merge friendly to me ;-) It would seem that 99% of all > documents and use cases could be treated with the same genericl merge > function. I think this idea is worth pursuing. Cheers Jan --