couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "anselmo silva" <anselmo....@gmail.com>
Subject Re: Bulk Load
Date Sun, 14 Sep 2008 01:08:49 GMT
Although i am totally new in CouchDB, i take the risk to comment on this
one. (somebody help me if i am wrong or not suficent explainable)

The kindness and strength of CouchDB, can change the paradigm how you use a
DB (persist data and access data).
There are things you must accept and others things you take with high
performance (map/reduce approach).

Revisions are not a out-of-the-box feature. CouchDB revisions exists for
database conflict management, and you lost it when you execute a compact (db
optimization).
If you want revisions, you must implement that application logic and
seralize it in CouchDB.
Two possivle ways are:
- A document for witch revision, with a revision (or version) field
(revision:2.34).
- Or a main document with a collection field [ ], where you include all the
revisions data(revisions:[{name:'abc', value:'123', revision:2.34}]).

In both cases i wouldnt borrow about the data grow,  that will happens in
any database you use ( if you want revisions ).

For reference follow jchris <http://github.com/jchris>, a great reference...

http://github.com/jchris/couchdb-reduce-example/tree/master

Cheers
Anselmo








On Sun, Sep 14, 2008 at 1:24 AM, Ronny Hanssen <super.ronny@gmail.com>wrote:

> Why is the revision control system in couchdb inadequate for, well,
> revision
> control? I thought that this feature indeed was a feature, not just an
> internal mechanism for resolving conflicts?
> Ronny
>
> 2008/9/14 Calum Miller <calum_miller@yahoo.com>
>
> > Hi Chris,
> >
> > Many thanks for your prompt response.
> >
> > Storing  a complete new version of each bond/instrument every day seems a
> > tad excessive. You can imagine how fast the database will grow overtime
> if a
> > unique version of each instrument must be saved, rather than just the
> > individual changes. This must be a common pattern, not confined to
> > investment banking. Any ideas how this pattern can be accommodated within
> > CouchDB?
> >
> > Calum Miller
> >
> >
> >
> >
> >
> > Chris Anderson wrote:
> >
> >> Calum,
> >>
> >> CouchDB should be easily able to handle this load.
> >>
> >> Please note that the built-in revision system is not designed for
> >> document history. Its sole purpose is to manage conflicting documents
> >> that result from edits done in separate copies of the DB, which are
> >> subsequently replicated into a single DB.
> >>
> >> If you allow CouchDB to create a new document for each daily import of
> >> each security, and create a view which makes these documents available
> >> by security and date, you should be able to access securities history
> >> fairly simply.
> >>
> >> Chris
> >>
> >> On Sat, Sep 13, 2008 at 12:31 PM, Calum Miller <calum_miller@yahoo.com>
> >> wrote:
> >>
> >>
> >>> Hi,
> >>>
> >>> I trying to evaluate CouchDB for use within investment banking, yes
> some
> >>> of
> >>> these banks still exist. I want to load 500,000 bonds into the database
> >>> with
> >>> each bond containing around 100 fields. I would be looking to bulk load
> a
> >>> similar amount of these bonds every day whilst maintaining a history
> via
> >>> the
> >>> revision feature. Are there any bulk load features available for
> CouchDB
> >>> and
> >>> any tips on how to manage regular loads of this volume?
> >>>
> >>> Many thanks in advance and best of luck with this project.
> >>>
> >>> Calum Miller
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
>



-- 
Anselmo Silva

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message