couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cottlehuber <d...@muse.net.nz>
Subject Re: Lookup views
Date Fri, 13 Jan 2012 20:46:47 GMT
On 13 January 2012 12:56, Nils Breunese <N.Breunese@vpro.nl> wrote:
> Matt <matthew-m.rose@ubs.com> wrote:
>
>> Suppose we buy an sell an awful lot of fruit, mostly apples and oranges but sometimes
bananas, pears and anything else that shoppers want. We take a feed (from another system)
of every trade and load them all into couch as we go along, then define a view to aggregate
them up by various different aspects of the sale. We can control the grouping level to produce
many different reports out the same view and it works very well for us. Some of the attributes
on the trades documents are not on the original feed so we enrich the data before PUTting
them in couch. This is also fine.
>>
>> Unfortunately, sometimes the enrichment changes. For example each trade has a fruit
field and we have a mapping from fruit to fruit type. Today apples, oranges and pears are
classified as round fruit whereas bananas are long fruit so we can put a fruit type field
on the document as we load it in but tomorrow that classification might change and pears might
become lumpy fruit instead of round fruit. We could go back and update every pear trade in
the database but we'd rather not have fruit type on the document at all and maintain a separate
mapping "table".
>>
>> We thought that perhaps the map function could reference another database that was
keyed on fruit and had documents containing the fruit type but don't seem to be able to perform
this lookup in the mapping. We could get the document itself on the result but not extract
just one field to augment the original documents. We also considered generating the view's
design dynamically out of a mapping table and replicating the mapping into explicit logic
in the javascript but this seemed like an abuse.
>>
>> In short, how would you implement a pre-aggregation lookup table?
>
> I think I'd just put the lookup table in the map function. Another option is indeed to
update the documents and change the fruit type. Which of these two yields best performance
probably depends on how large your lookup table would be and how many documents you'd have
to update. Map functions need to be completely standalone in order to ensure things keep working
when you start doing things like distributing your data, so calls to external stuff is a no-go.
>
> Nils.
> ------------------------------------------------------------------------
>  VPRO   www.vpro.nl
> ------------------------------------------------------------------------

Its worth noting that changing the map functions in either way will
require the view to be rebuilt before it can be used which might be an
expensive or timeconsuming operation. This is a common workaround for
reducing the operational impact of doing that
http://wiki.apache.org/couchdb/How_to_deploy_view_changes_in_a_live_environment

I'd not update the docs directly as this will increase the DB size,
requiring more frequent compaction. In comparison a view generation
would likely be less expensive.

A+
Dave

Mime
View raw message