Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <5252D880.6060400@fiset.ca>
Date: Mon, 07 Oct 2013 11:51:28 -0400
From: Jean-Pierre Fiset <jp@fiset.ca>
User-Agent: Mozilla/5.0 (X11; Linux i686;
 rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: user@couchdb.apache.org
Subject: Re: Contribution: CouchDb dump and reload
References: <524DB903.5010405@fiset.ca> <524DBABA.8080903@orgmeta.com>
 <9292ECFF-41C2-498C-A062-5FB6775C8B1F@couchbase.com>
 <CAHdjipKGBLH=W=yorVYYvbswnKad3rUxqb9uPaLzBj27qLGdNg@mail.gmail.com>
 <46C56CF3-C72A-49E3-AAD1-9A5EF914C74B@programmazione.it>
In-Reply-To: <46C56CF3-C72A-49E3-AAD1-9A5EF914C74B@programmazione.it>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

The tool we have developed uses a strategy similar to all_docs?include_docs=true to dump. The
format on disk breaks the document into the top level attributes into different files to make it
easy for humans to understand/edit. The format is similar to that used by the python tool
"couchapp". Furthermore, the attachments are saved in their own files with some supporting
information to keep track of information such as content type.

The restore process is more elaborate. Here are some features:

- When restoring, the restore tool ensures that a document modified since the last dump is not
replaced by the document on disk. This ensures that the documents located in the database
remains the authoritative ones. On a restore, an operator can force the restoration, but by
default it attempts to protect the database.

- It is possible to restore only specific documents. This currently done by specifying the
identifiers of the documents to be restored.

- When uploading a documents, all the changes are applied at once, increasing the document
revision only by one.

- Disk documents that are equivalent to the ones found in the database are detected and the
upload is skipped.

The only "proprietary" facet of this process are a digest installed on the database document to
keep track of the document's content, the name of the attribute to save the digest and the
method the digest is computed. Every time a document is uploaded using the restore tool, a
digest of the document is calculated and added to the document. This allows the restore tool to
find out whether a database document has been manually modified since the last time it was
uploaded. Since it would be difficult for someone to compute the digest, or craft a new version
document to collide with the current digest, it ensures that the restore tool does not
inadvertantly overwrite changes to the database perform by a human.

The method to compute the digest is straight forward and could be standardized to allow various
tools to interact with a single database.

As far as standardizing how documents should be stored to disk, it would probably be a
worthwhile endeavour.

JP

On 2013-10-03 20:28, Filippo Fadda wrote:
> Is it basically an all_docs + include_docs?
> 
> -Filippo
> 
> On Oct 3, 2013, at 11:51 PM, Alexander Shorin wrote:
> 
>> On Fri, Oct 4, 2013 at 1:28 AM, Jens Alfke <jens@couchbase.com> wrote:
>>> On Oct 3, 2013, at 11:43 AM, Vivek Pathak <vpathak@orgmeta.com> wrote:
>>>
>>>> Just fyi,  there is couchdb-dump available in
>>>> tihttp://code.google.com/p/couchdb-python/
>>>
>>> Looks like these two tools use entirely different data formats. Has anyone thought of defining a common format for database dumps?
>>
>> I think it will be hard to define such.
>>
>> Dumping CouchDB data as JSON looks intuitive and requires less
>> additional actions for import/export. Having couchdb-python approach
>> with multipart format provides lesser footprint, but requires a more
>> tricky processing (boundaries, headers). Both solutions may use
>> CouchDB API without any additional data conversion.  And both requires
>> a lot of disk space, much more than if you just copy database file or
>> make a replica of it, unless you xz-zip the output.
>>
>> --
>> ,,,^..^,,,
>