couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Woodhead <simon.woodh...@simwood.com>
Subject Hello / Archiving
Date Sat, 31 Jul 2010 17:45:01 GMT
Hi folks,

First off: thanks! CouchDB is something I found out about by accident by
virtue of my g/f breaking her toe but that's another story, although I'm
really glad she did. I've been reading about it lots and have finally
deployed it in a real-world trial today.

We store gazillions of small records. We already have them in JSON as they
flow through our queuing system. Along the way we take key bits of that info
into RDMS for main use but the end result is we need to keep the rest
somewhere for easy access. We tried S3 but it was too slow, we tried
simpleDB but hit the limits in hours so we've finally bit the bullet and are
trying CouchDB. So far so good - we write with our unique id as the _id
which make retrieval superbly easy. We haven't got into views but don't need
to for this application - just lots of parallel writes and a very occasional
retrieval.

However, in a little over an hour of the trial we have a 2GB database and it
is growing quickly. This is no great surprise as there is an awful lot of
data getting pumped in here - in raw JSON it amounts to 10-15GB per day. So
my question to the list is are there any approved methods of archiving?
Sharding seems unnecessary since one node more than handles our read/write
requirements, but it is going to need a tonne of storage. We'll also be
replicating between sites so the total requirement will be doubled.
Presently it is running as a VM with storage on the SAN so any usage is
expensive.

One idea I have had is to name the database to include the date and then
databases above a certain age could be detached and compressed somewhere.
Does that sound workable and is there an approved method for detaching? It
looks like we could just move the file without any adverse consequences but
I wanted to check! And re-attaching?

So far, CouchDB is looking like a dream come true and I'm very sure we're
going to move other applications to it as we find our way around. I have no
doubt we're going to be a relatively large deployment when we find our feet
with it so thanks again to all involved.

cheers,
Simon





-- 
Simon Woodhead FCSI
Managing Director
<http://www.simwood.com>
Simwood eSMS Limited
Wholesale Telecommunications

Keep up with the latest news from Simwood:
<http://feeds.simwood.com/SimwoodNews>
<http://www.facebook.com/pages/Simwood-eSMS-Limited/146897445321268>
  <http://twitter.com/simwoodesms>
<http://twitter.com/simwoodesms>
w: http://www.simwood.com

--
***** Email confidentiality notice *****

This message is private and confidential. If you have received this message in error, please
notify us and remove it from your system.


Simwood eSMS Limited is a limited company registered in England and Wales. Registered number:
03379831. Registered office: c/o HW Chartered Accountants, Keepers Lane, The Wergs, Wolverhampton,
WV6 8UA. Trading address: Falcon Drive, Cardiff Bay, Cardiff, CF10 4RU.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message