Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of dionne@dionne-associates.com
 designates 67.222.38.31 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default;
 d=dionne-associates.com;
	h=Received:Message-Id:From:To:In-Reply-To:Content-Type:Content-Transfer-Encoding:Mime-Version:Subject:Date:References:X-Mailer:X-Identified-User;
	b=OrHPnItjAt0UzHNowlXb/eyhJkM7x4/NBpyU40oT+d5UNDyyLzCztGT6UUUvGrlXlUBHDJt4P/O7+8oWZL1aE5e6YueChPRjt+SXsx8q+6ikhJ5fivTz3cdXQAhtVonA;
Message-Id: <EED13D38-62A4-409B-93E7-C5B29DBA554B@dionne-associates.com>
From: Robert Dionne <dionne@dionne-associates.com>
To: dev@couchdb.apache.org
In-Reply-To: <BBBDDD93-2F02-4BCD-86E9-A85E13E56089@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: History Proposal
Date: Mon, 3 Aug 2009 12:17:39 -0400
References: <FBDEA366-1DF6-47A2-A67D-50BAE7A15F4D@jasondavies.com>
 <b7cd8ed10907310642l7dc3e33dj5ccda198ec0d6f32@mail.gmail.com>
 <B78FFF6C-6289-4C06-BF11-A8F93F1312E2@jasondavies.com>
 <e282921e0908021229s5d4329e0o80d20fb5a399b65f@mail.gmail.com>
 <CF6C38C6-F913-4358-A6DB-4EC9BBBE1EFE@apache.org>
 <EF91626A-118E-4C9A-97C2-C3DF9ACAFD3F@jasondavies.com>
 <BBBDDD93-2F02-4BCD-86E9-A85E13E56089@apache.org>


On Aug 3, 2009, at 12:09 PM, Damien Katz wrote:

>
> On Aug 3, 2009, at 11:55 AM, Jason Davies wrote:
>
>> Hi Damien,
>>
>> On 3 Aug 2009, at 16:39, Damien Katz wrote:
>>
>>> On Aug 2, 2009, at 3:29 PM, Chris Anderson wrote:
>>>
>>>> On Sat, Aug 1, 2009 at 3:29 AM, Jason  
>>>> Davies<jason@jasondavies.com> wrote:
>>>>>
>>>>> On 31 Jul 2009, at 14:42, Benoit Chesneau wrote:
>>>>>
>>>>>> 2009/7/31 Jason Davies <jason@jasondavies.com>:
>>>>>>
>>>>>>> The main points of this proposal are:
>>>>>>>
>>>>>>> 1. Store the historical versions of documents in a separate  
>>>>>>> database.
>>>>>>> This
>>>>>>> is for a number of reasons: a) keeping it separate means we  
>>>>>>> don't clog up
>>>>>>> the main database with historical data b) history-specific  
>>>>>>> views can be
>>>>>>> kept
>>>>>>> here c) non-intrusive implementation of this is easier.
>>>>>>> 2. The change will be made at the couch_db layer so that *any*  
>>>>>>> change to
>>>>>>> any
>>>>>>> document in the target database will be mirrored to the  
>>>>>>> history database.
>>>>>>
>>>>>> seem good.
>>>>>>
>>>>>>> 3. Each and every change to a document will result in a new  
>>>>>>> document
>>>>>>> being
>>>>>>> created in the history database (with a new ID) containing an  
>>>>>>> exact copy
>>>>>>> of
>>>>>>> that document e.g. {_id: <new ID>, doc: <exact copy of doc> }.
>>>>>>
>>>>>> How would you handle case of attachements ? If attachements are  
>>>>>> copied
>>>>>> for each revision of a doc, it would take a lot of place. Maybe
>>>>>> storing attachements in their own doc could be solution though.  
>>>>>> So
>>>>>> storing a revision would be
>>>>>>
>>>>>> store attachements in differents docs
>>>>>> create a doc  {_id: <id>, doc: <doc>, attachments: [<id1>, ...]}
>>>>>>
>>>>>> attachements will be tests across revisions depending of their  
>>>>>> signature
>>>>>> if signature change, a new atatchment doc is created.
>>>>>>
>>>>>> Just a thought anyway.
>>>>>
>>>>> Good idea, the disk space issue would be quite important for  
>>>>> larger
>>>>> databases with larger number of changes.  I wonder if some kind of
>>>>> alternative storage layer supporting diffs would help here.   
>>>>> Probably
>>>>> something to consider as a future improvement.
>>>>>
>>>>>>
>>>>>>
>>>>>>> 4. Adding meta-data to changes can be handled by a custom  
>>>>>>> _update handler
>>>>>>> (yet to be developed) to set fields such as "last_modified" and
>>>>>>> "last_modified_user".
>>>>
>>>> I've been quiet on this thread as I'm largely in agreement with  
>>>> the proposal.
>>>>
>>>> I think the best route for implementation is to allow Erlang  
>>>> callbacks
>>>> on changes. This way we can write a simple history function that
>>>> copies off each change to a backup db, setting timestamps and  
>>>> userCtx
>>>> metadata on the way.
>>>>
>>>> The user interface could surface this function's activation in the
>>>> node config as a check box, and applications wouldn't need to know
>>>> about it at all. It should be possible to develop a generic futon- 
>>>> like
>>>> interface for browsing old documents to revert individual  
>>>> changes, so
>>>> users can work with non-backup-aware applications.
>>>>
>>>> As far as keeping track of time ranges when backups are turned off,
>>>> the user interface could record a timestamped metadata document  
>>>> to the
>>>> backup db whenever the switch is flipped.
>>>
>>> Some comments about the proposal
>>>
>>> 1.  The callbacks must be synchronous. Queueing them for writing  
>>> later means the queue can get overloaded and changes lost.
>>> 2 Changes can still get lost. We don't have commits across dbs, so  
>>> it's possible a crash during update will put the main and history  
>>> dbs out of sync.
>>> 3. Replicated changes get lost. If a client makes 5 edits to local  
>>> replica of a document, then replicates it to a server db, only the  
>>> most recent change get recorded in the history.
>>>
>>> I would prefer to store the history as attachments to the main  
>>> document.
>>
>> Can you expand on your last sentence in a bit more detail?  I  
>> assume you mean you would rather each document in the history db  
>> mirrored each document in the target db, with attachments storing  
>> historical versions?
>
> No, I mean the earlier revisions of the document, stored as  
> attachments to the current revision.

This seems like a simple approach. If history is enabled the  
attachments could be generated when a document is updated, or lazily  
at compaction time? I'm not sure how delete would be handled.

>
> The history then replicates with the document, and is always  
> available.
>
>>
>> To solve #3 we could also allow the history database to be  
>> replicated for use-cases where the entire history is desirable on  
>> all peers.
>
> The problems with the history database is there are a lot of edge  
> cases where the history gets out of sync, especially with  
> distributed edits. The system breaks easily in the face of network  
> and security errors.
>
> -Damien
>
>>
>> --
>> Jason Davies
>>
>> www.jasondavies.com
>>
>