couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Bronson <jabron...@gmail.com>
Subject Re: Multipart MIME in dump tool
Date Wed, 17 Jun 2009 16:51:24 GMT
To revive an old thread:
I needed a script to dump a large (>30G) couchdb database on a nightly basis
for backup purposes, to be performed while couchdb is running, and
noticed couchdb-python
issue 58 <http://code.google.com/p/couchdb-python/issues/detail?id=58>:
couchdb-dump
fails if size of dumpfile > memory. I looked at the scripts attached in the
comments and realized that I had similar but different needs: I just needed
to stream the response to _all_docs_by_seq?include_docs=true directly to
stdout, without doing any json encoding along the way. (The complementary
load does have to do json encoding (for instance to check the "deleted":true
flag), but this is fine.) I thought I would share the script with the
community in case it's helpful to anyone else as well as to solicit
feedback:

https://svn.openplans.org/melk/util/streamcouch.py



Here is the usage:

Usage: ./streamcouch.py [dump | load] DBURL

"dump" requests "_all_docs_by_seq?include_docs=true" from DBURL
and streams the response to stdout.

"load" creates a database at DBURL with documents read from stdin
in the format output by "_all_docs_by_seq?include_docs=true".
Requires couchdb-python <http://code.google.com/p/couchdb-python/>.

Ex.
  ./streamcouch.py dump http://localhost:5984/backedup > dump
  ./streamcouch.py load http://localhost:5984/restored < dump



I have tested a dump / load roundtrip locally and it
worked. I should note that none of our documents have inline attachments.
There was talk earlier in this thread that documents with attachments cannot
be posted with a _rev. It would be trivial to remove the _rev during the
load, I just haven't needed to.

I should also note that my next version of the script will allow you to
perform incremental backups by passing a startkey to _all_docs_by_seq. The
complementary load will be able accept multiple responses to
_all_docs_by_seq on stdin.

Looking forward to your responses.


On Tue, Apr 7, 2009 at 8:49 AM, Matt Goodall <matt.goodall@gmail.com> wrote:

>
> 2009/4/7 Jeff Hinrichs - DM&T <dundeemt@gmail.com>:
> >
> >
> > On Tue, Apr 7, 2009 at 4:37 AM, Matt Goodall <matt.goodall@gmail.com>
> wrote:
> >>
> >> 2009/4/7 Jeff Hinrichs - DM&T <dundeemt@gmail.com>:
> >> >
> >> >
> >> > On Mon, Apr 6, 2009 at 2:03 PM, Matt Goodall <matt.goodall@gmail.com>
> >> > wrote:
> >> >>
> >> >> 2009/4/6 Matt Goodall <matt.goodall@gmail.com>:
> >> >> > 2009/4/5 Jeff Hinrichs - DM&T <dundeemt@gmail.com>:
> >> >> >>> So personally, for now, I would write the dump/load tools
using
> >> >> >>> plain
> >> >> >>> old httplib from the standard library. It's more than
capable.
> The
> >> >> >>> only bit that might involve decoding from JSON to Python
is
> >> >> >>> removing
> >> >> >>> the _rev; everything else is a matter of streaming data
from HTTP
> >> >> >>> to
> >> >> >>> disk and vice-versa.
> >> >> >>
> >> >> >> Although that is done currently, I don't believe that it is
> strictly
> >> >> >> required by couchdb.  In fact I just tested and you can insert
> >> >> >> (while
> >> >> >> loading to an empty database) with the _rev property containing
> >> >> >> information.  The _rev is then updated and the resulting _rev
> >> >> >> returned.
> >> >> >> So
> >> >> >> there is no need to remove the _rev when loading a dump file.
> >> >> >
> >> >> > Unfortunately, you do still have to remove the _rev for those
> >> >> > documents with inline attachments, otherwise you get a conflict
> error
> >> >> > from couchdb. I don't know if it's a bug to create a document
with
> a
> >> >> > _rev or if the _rev + inline attachments is just an inconsistency.
> >> >> > I'll ask on the couchdb list later and post back here.
> >> >>
> >> >> Accepting a new document containing a _rev appears to be a bug in
> >> >> CouchDB. So, it definitely needs removing which is arguably the
> >> >> correct thing to do anyway.
> >> >
> >> >
> >> >>
> >> >> - Matt
> >> >
> >> > Matt,
> >> > Saw your post and then reviewed my test scripts only to realize that
> you
> >> > are
> >> > correct about couchdb freaking out with _rev+attachment.  However, I
> >> > don't
> >> > agree with Damien about which is the bug.  I see his point of view --
> he
> >> > is
> >> > seeing this as standard couchdb operation.
> >> >
> >> > Quite frankly, to be a proper dump/load mechanism -- you should be
> able
> >> > to
> >> > dump dbA, then create a dbB, then load the dump from dbA into dbB and
> >> > when
> >> > you replicate from one to the other, they should appear to be already
> >> > synchronized (no replication events occur)  If a dump load cycle
> causes
> >> > dbA
> >> > to transform into dbA' then it is not a dump/load -- it's a fetch and
> >> > Insert.
> >>
> >> Ah, I think that's a different sort of dump/load than couchdb-python
> >> provides. I see couchdb-python's dump/load is a sort of
> >> snapshot/bootstrap tool. You're talking about a disconnected
> >> replication process. Both probably have their uses.
> >
> >
> > Now that you put it that way, yes.  I was using replication to
> demonstrate
> > that a database reloaded from a dump should equal the originally dumped
> > database.  Replication is inherent w/ couchdb, so if a reloaded database
> > will respond differently to replication than the original, something is
> > lost.
> >
> > Couchdb essentially uses the _id + _rev as a unique index to the data.
> The
> > inability to recreate that unique index is the problem that, I think,
> needs
> > to be corrected.
> >
> >>
> >> >
> >> > couchdb proper really needs to correct this situation.  Dump and load
> >> > needs
> >> > to put couchdb into a different mode so that this can be accomplished.
> >> > i.e.
> >> > /database/_dump -- which would dump json documents out  and then a
> >> > /database/_load where you would post the contents of the _dump to
> reload
> >> > a
> >> > database.  And I mean load -- not insert.
> >>
> >> I'm sure this can already be achieved using HTTP but, AFAICT, it's not
> >> fully documented yet.
> >
> > There is a thread on couchdb-dev,
> >
> http://mail-archives.apache.org/mod_mbox/couchdb-dev/200904.mbox/%3C9AF648C2-0503-48E8-A335-13BAEB9B1B3D@apache.org%3E
> > that appears to be talking about such a method.  However, it would
> require
> > the _rev as a parameter instead of part of the json document.
> >
> >>
> >> And yes, it might be nice for CouchDB to provide support for a
> >> disconnected replication mode (i.e. replicate to disk) that assumes
> >> the data will be loaded into an empty database (or at least a database
> >> that has not seen those documents before). However, wouldn't that dump
> >> basically be a compacted database?
> >
> > It would be, but currently, none of the scripts (haven't looked at your
> > yet), request conflicted.  I haven't written the test for this yet, but
> my
> > analysis is that only the current "winning" _rev of the data.  A
> dump/load
> > cycle currently looses that information, if present.  It has not bitten
> me
> > yet, but I can see it being a problem.
>
> My version is as lossy as the original couchdb-dump/load ... just a
> couple of orders of magnitude more efficient ;-).
>
> >
> >>
> >> >
> >> > There are times when you need to be able to dump/load a database.
> Some
> >> > times for error recovery, some times for debugging and some times for
> >> > legal
> >> > reasons but without a proper couchdb api for it, we are whistling in
> the
> >> > wind.
> >>
> >> That already works - just copy the .couch file for your database.
> >> CouchDB's append-only model should mean you never get a copy of a
> >> partly written database.
> >
> > You do have a point.  However, that .couch file is dependent on the
> running
> > version of couchdb - so that would need to be backed up too.  Maybe I am
> > still need to shake off the dust of current RDBMS but all of them support
> > the idea of  dump/load.  Something just feels very 80'ish about backing
> up
> > the actual data file.
>
> You're right, there should be a version-independent format and it
> should be part of the CouchDB distribution.
>
> Every time a .couch file format-breaking change is made you have to
> replicate from to a new CouchDB server. For the 0.9 release where the
> .couch file format *and* the replication stream format changed not
> even that was possible and someone (jchris, I think) wrote a script to
> help.
>
> >
> >>
> >> Note: you can actually perform a disconnected replication using a
> >> copy: copy a .couch file to a new CouchDB instance giving it a
> >> temporary database name, replicate from the temporary database to the
> >> real database, delete the temporary database. Not especially nice, but
> >> not too bad either. All CouchDB needs to do to improve that is to
> >> provide a way to replicate from a file on disk rather than a locally
> >> installed database.
> >
> > Agreed,  and then this whole thread becomes a bike shed. ;)
>
> Still some use, I think. I still think the ability to bulk load a
> bunch of documents from a simple file is useful. For instance, when
> setting up a new database it's not uncommon to need to bootstrap the
> data in it before pointing an web server at it.
>
> >
> >
> > Regards,
> >
> > Jeff Hinrichs
> >>
> >> - Matt
> >>
> >>
> >
> >
> >
> > --
> > Jeff Hinrichs
> > Dundee Media & Technology, Inc
> > jeffh@dundeemt.com
> > 402.218.1473
> > web: www.dundeemt.com
> > blog: inre.dundeemt.com
> >
> > >
> >
>
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups
> "CouchDB-Python" group.
> To post to this group, send email to couchdb-python@googlegroups.com
> To unsubscribe from this group, send email to
> couchdb-python+unsubscribe@googlegroups.com<couchdb-python%2Bunsubscribe@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com/group/couchdb-python?hl=en
> -~----------~----~----~----~------~----~------~--~---
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message