couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J Chris Anderson <jch...@apache.org>
Subject Re: Map/reduce and transactions
Date Fri, 13 Aug 2010 15:07:10 GMT

On Aug 13, 2010, at 7:55 AM, Simon Woodhead wrote:

> Hi folks
> 
> As we explore more ways to migrate to CouchDB we're exploring alternatives to transactions.
The books shows a bank where transfers are stored as a single document with the balance being
the result of a map/reduce function. That makes sense. 
> 
> For us, our scale is tipped differently in that we have hundreds of millions of tiny
transactions affecting relatively few balances. In MySQL (ironically) we denormalise this
and hold balances in their own table but then can insert transactions and update balances
within a single transaction. 
> 
> Looking at moving this to CouchDB would mean getting rid of the balances table and just
using a map/reduce function. I recognise that a given document will only be handled once and
that this is therefore more efficient than it may seem to a SQL jock like me but I wanted
to ask about whether it truly scales at volume. 
> 
> I guess I'm asking whether whatever index the view creates contains a reference to every
document (and thus gets bigger with more documents) or just contains the output and the _id
of the last document processed. I can see the first one running into issues quickly whilst
the second would seem to scale indefinitely. 
> 
> FWIW it takes over a day to compute the sum against the table in MySQL as the table is
being constantly appended - hence why we denormalised it! It therefore feels really strange
to normalise more when moving to a document store so any advice is welcome! 

The index in the banking example in the book will indeed grow as transactions happen. If you
have a very high rate of transactions, you might want to do like banks do, and "close" each
month, by moving new transactions to a new database file at the end of the month. Then you
can do a transaction for each account to carry forward the closing balance at the end of the
last month, to the current month.

After X months, you can throw away the old transaction databases, probably storing the monthly
rollups somewhere for posterity.

If you rate is high enough, substitute days or hours for months.

> 
> Thanks
> Simon--
> ***** Email confidentiality notice *****
> 
> This message is private and confidential. If you have received this message in error,
please notify us and remove it from your system.
> 
> 
> Simwood eSMS Limited is a limited company registered in England and Wales. Registered
number: 03379831. Registered office: c/o HW Chartered Accountants, Keepers Lane, The Wergs,
Wolverhampton, WV6 8UA. Trading address: Falcon Drive, Cardiff Bay, Cardiff, CF10 4RU.
> 
> 


Mime
View raw message