couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yue Chuan Lim <shasder...@gmail.com>
Subject Re: Hello / Archiving
Date Sat, 31 Jul 2010 17:56:35 GMT
Ditto, but for some reason, compacting the database (and views (and cleaning
up the views)) does slim my database down quite a bit. Havn't looked into
the reason why that happens though.

Would appreciate enlightenment on this point. :)

Cheers

On Sun, Aug 1, 2010 at 1:51 AM, Simon Woodhead
<simon.woodhead@simwood.com>wrote:

> Hi Yue,
>
> Blimey, that was quick :-)
>
> Our data is single revision and no deletes so as I understand it compacting
> wouldn't do much as there's nothing to clear out. We have tried though and
> sure enough, it didn't! We'll try it when it has been running for longer.
>
> cheers,
> Simon
>
>
> On Sat, Jul 31, 2010 at 6:48 PM, Yue Chuan Lim <shasderias@gmail.com>
> wrote:
>
> > Have you tried compacting the database? (:
> >
> > My current use case is similar, I pretty much need to run compaction once
> a
> > day (from my estimates).
> >
> > Cheers
> >
> > On Sun, Aug 1, 2010 at 1:45 AM, Simon Woodhead
> > <simon.woodhead@simwood.com>wrote:
> >
> > > Hi folks,
> > >
> > > First off: thanks! CouchDB is something I found out about by accident
> by
> > > virtue of my g/f breaking her toe but that's another story, although
> I'm
> > > really glad she did. I've been reading about it lots and have finally
> > > deployed it in a real-world trial today.
> > >
> > > We store gazillions of small records. We already have them in JSON as
> > they
> > > flow through our queuing system. Along the way we take key bits of that
> > > info
> > > into RDMS for main use but the end result is we need to keep the rest
> > > somewhere for easy access. We tried S3 but it was too slow, we tried
> > > simpleDB but hit the limits in hours so we've finally bit the bullet
> and
> > > are
> > > trying CouchDB. So far so good - we write with our unique id as the _id
> > > which make retrieval superbly easy. We haven't got into views but don't
> > > need
> > > to for this application - just lots of parallel writes and a very
> > > occasional
> > > retrieval.
> > >
> > > However, in a little over an hour of the trial we have a 2GB database
> and
> > > it
> > > is growing quickly. This is no great surprise as there is an awful lot
> of
> > > data getting pumped in here - in raw JSON it amounts to 10-15GB per
> day.
> > So
> > > my question to the list is are there any approved methods of archiving?
> > > Sharding seems unnecessary since one node more than handles our
> > read/write
> > > requirements, but it is going to need a tonne of storage. We'll also be
> > > replicating between sites so the total requirement will be doubled.
> > > Presently it is running as a VM with storage on the SAN so any usage is
> > > expensive.
> > >
> > > One idea I have had is to name the database to include the date and
> then
> > > databases above a certain age could be detached and compressed
> somewhere.
> > > Does that sound workable and is there an approved method for detaching?
> > It
> > > looks like we could just move the file without any adverse consequences
> > but
> > > I wanted to check! And re-attaching?
> > >
> > > So far, CouchDB is looking like a dream come true and I'm very sure
> we're
> > > going to move other applications to it as we find our way around. I
> have
> > no
> > > doubt we're going to be a relatively large deployment when we find our
> > feet
> > > with it so thanks again to all involved.
> > >
> > > cheers,
> > > Simon
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Simon Woodhead FCSI
> > > Managing Director
> > > <http://www.simwood.com>
> > > Simwood eSMS Limited
> > > Wholesale Telecommunications
> > >
> > > Keep up with the latest news from Simwood:
> > > <http://feeds.simwood.com/SimwoodNews>
> > > <http://www.facebook.com/pages/Simwood-eSMS-Limited/146897445321268>
> > >  <http://twitter.com/simwoodesms>
> > > <http://twitter.com/simwoodesms>
> > > w: http://www.simwood.com
> > >
> > > --
> > > ***** Email confidentiality notice *****
> > >
> > > This message is private and confidential. If you have received this
> > message
> > > in error, please notify us and remove it from your system.
> > >
> > >
> > > Simwood eSMS Limited is a limited company registered in England and
> > Wales.
> > > Registered number: 03379831. Registered office: c/o HW Chartered
> > > Accountants, Keepers Lane, The Wergs, Wolverhampton, WV6 8UA. Trading
> > > address: Falcon Drive, Cardiff Bay, Cardiff, CF10 4RU.
> > >
> > >
> >
>
> --
> ***** Email confidentiality notice *****
>
> This message is private and confidential. If you have received this message
> in error, please notify us and remove it from your system.
>
>
> Simwood eSMS Limited is a limited company registered in England and Wales.
> Registered number: 03379831. Registered office: c/o HW Chartered
> Accountants, Keepers Lane, The Wergs, Wolverhampton, WV6 8UA. Trading
> address: Falcon Drive, Cardiff Bay, Cardiff, CF10 4RU.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message