couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikeal Rogers <mikeal.rog...@gmail.com>
Subject Re: Hello / Archiving
Date Sat, 31 Jul 2010 17:54:51 GMT
Yeah, all of the CouchDB db and view files are append-only so they are
*always* in a consistent state and can be started up at any time.

Copying and moving the db files somewhere will be fine and all you need to
do to start them back up is move them back :) CouchDB kinda rocks like this.

-Mikeal

On Sat, Jul 31, 2010 at 5:48 PM, J Chris Anderson <jchris@gmail.com> wrote:

>
> On Jul 31, 2010, at 10:45 AM, Simon Woodhead wrote:
>
> > Hi folks,
> >
> > First off: thanks! CouchDB is something I found out about by accident by
> > virtue of my g/f breaking her toe but that's another story, although I'm
> > really glad she did. I've been reading about it lots and have finally
> > deployed it in a real-world trial today.
> >
> > We store gazillions of small records. We already have them in JSON as
> they
> > flow through our queuing system. Along the way we take key bits of that
> info
> > into RDMS for main use but the end result is we need to keep the rest
> > somewhere for easy access. We tried S3 but it was too slow, we tried
> > simpleDB but hit the limits in hours so we've finally bit the bullet and
> are
> > trying CouchDB. So far so good - we write with our unique id as the _id
> > which make retrieval superbly easy. We haven't got into views but don't
> need
> > to for this application - just lots of parallel writes and a very
> occasional
> > retrieval.
> >
> > However, in a little over an hour of the trial we have a 2GB database and
> it
> > is growing quickly. This is no great surprise as there is an awful lot of
> > data getting pumped in here - in raw JSON it amounts to 10-15GB per day.
> So
> > my question to the list is are there any approved methods of archiving?
> > Sharding seems unnecessary since one node more than handles our
> read/write
> > requirements, but it is going to need a tonne of storage. We'll also be
> > replicating between sites so the total requirement will be doubled.
> > Presently it is running as a VM with storage on the SAN so any usage is
> > expensive.
> >
> > One idea I have had is to name the database to include the date and then
> > databases above a certain age could be detached and compressed somewhere.
> > Does that sound workable and is there an approved method for detaching?
> It
> > looks like we could just move the file without any adverse consequences
> but
> > I wanted to check! And re-attaching?
> >
>
> this is what I would suggest.
>
> the 1.x line will be binary compatible, so you should have no trouble
> re-activating stored databases.
>
> > So far, CouchDB is looking like a dream come true and I'm very sure we're
> > going to move other applications to it as we find our way around. I have
> no
> > doubt we're going to be a relatively large deployment when we find our
> feet
> > with it so thanks again to all involved.
> >
>
> if you hit walls, please ask here before getting frustrated. there is a lot
> of experience with very large scale CouchDB, so folks will be able to help.
>
> Chris
>
> > cheers,
> > Simon
> >
> >
> >
> >
> >
> > --
> > Simon Woodhead FCSI
> > Managing Director
> > <http://www.simwood.com>
> > Simwood eSMS Limited
> > Wholesale Telecommunications
> >
> > Keep up with the latest news from Simwood:
> > <http://feeds.simwood.com/SimwoodNews>
> > <http://www.facebook.com/pages/Simwood-eSMS-Limited/146897445321268>
> >  <http://twitter.com/simwoodesms>
> > <http://twitter.com/simwoodesms>
> > w: http://www.simwood.com
> >
> > --
> > ***** Email confidentiality notice *****
> >
> > This message is private and confidential. If you have received this
> message in error, please notify us and remove it from your system.
> >
> >
> > Simwood eSMS Limited is a limited company registered in England and
> Wales. Registered number: 03379831. Registered office: c/o HW Chartered
> Accountants, Keepers Lane, The Wergs, Wolverhampton, WV6 8UA. Trading
> address: Falcon Drive, Cardiff Bay, Cardiff, CF10 4RU.
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message