couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Design doc index switching
Date Fri, 17 May 2019 09:59:03 GMT


> On 16. May 2019, at 23:03, Robert Samuel Newson <rnewson@apache.org> wrote:
> 
> I suggest an alternative; the new design document could include the _id of design document
it’s replacing (“_replaces”:”_design/foo”). On completion of the view build of the
new design document, CouchDB itself updates the named _id to the same content as the new design
document (strictly, only the parts needed to make the view sig match) (perhaps it also deletes
the new document).
> 
> The advantage to this is that queries to the original design document continue to work
throughout and at no point is there a discrepancy between the design documents contents (the
map and reduce functions, etc) and the results you get from it.


As Stefan points out, this only works if the consuming code knows how to handle both index
result formats. If not, we need a way for either CouchDB to signal the build being ready or
a way to confirm the swap to coincide with a code deploy.


> 
> B.
> 
>> On 16 May 2019, at 14:55, Jan Lehnardt <jan@apache.org> wrote:
>> 
>> +1 on solving this for all users, and same caveats as Stefan raises :)
>> 
>>> On 16. May 2019, at 09:38, Stefan du Fresne <stefan@medicmobile.org> wrote:
>>> 
>>> Hey Garren,
>>> 
>>> Having this a native part of CouchDB seems like a really cool idea: we have automated
the manual dance you're talking about with our deployment tooling, but it would be really
nice not to have to!
>>> 
>>> I'm not clear how it would work though, at least in terms of coherent deployments.
View changes are, like SQL migrations, an often non-backwards compatible change that has to
occur as your new code deploys.
>>> 
>>> Currently the naive approach is you deploy your new code alongside design doc
changes, which then block view queries on first request until they're ready to go.
>>> 
>>> The better approach is what you describe, which is what we do now, where we extract
our design documents out of our deployment bundle and place them in a "staging" location to
allow them to warm, then rename them and do the actual code deployment once that's complete
(managed by an external deployment service we built). This importantly lets us split the "warming"
bit from the deployment bit: we only deploy new code once the design documents that are shipped
with that code is ready to go.
>>> 
>>> How would you foresee this kind of flow happening here? Would there be a way
to query the design doc to know if it had flipped to the new version yet? Would you be able
to control when this flip occurs? Or would the expectation be that your code handles both
versions gracefully?
>>> 
>>> As an example to mull over, let's say you have design doc v1, which has view
a. You push design doc v2, which has added view b, but has also changed view a in some backwards
incompatible way. While v2 is still building and is not yet the active doc:
>>> - If you queried view a you'd get the v1 version, that's clear
>>> - If you queried view b you'd get... a 404? Some other custom code?
>>> - If you GET the design document what doc would you see? Presumably v2?
>>> - Could you query something to determine which version is currently active? Or
perhaps just whether there is a background version building at all?
>>> 
>>> Cheers,
>>> Stefan
>>> 
>>>> On 16 May 2019, at 07:51, Garren Smith <garren@apache.org> wrote:
>>>> 
>>>> Hi Everyone,
>>>> 
>>>> A common pattern we see for updating large indexes that can take a few days
>>>> to build, is create a new design docs with the new updated views. Then once
>>>> the new design doc is built, a user changes the new design doc’s id to
the
>>>> old design doc. That way the CouchDB url for the views remain the same and
>>>> any requests to the design doc url automatically get the latest views only
>>>> once they built.
>>>> 
>>>> This is an effective way of managing building large indexes, but the
>>>> process is quite complicated and often users get it wrong. I would like to
>>>> propose that we move this process into CouchDB and let CouchDB handle the
>>>> actual process. From a users perspective, they would add a field to the
>>>> options of a design document that lets CouchDB know, that this build needs
>>>> to be built in the background and only replace the current index once its
>>>> built:
>>>> 
>>>> ```
>>>> {
>>>> "_id": "_design/design-doc-id",
>>>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
>>>> "views": {
>>>> "map-view": {
>>>>   "map": "function (doc) {\n  emit(doc._id, 1);\n}"
>>>> }
>>>> },
>>>> "language": "javascript",
>>>> "options": {
>>>>     "build_and_replace": true
>>>> }
>>>> }
>>>> ```
>>>> 
>>>> I think this is something we could build quite effectively once we have
>>>> CouchDB running on top of FoundationDB. I don’t want to implement it for
>>>> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
>>>> we build out the map/reduce indexes.
>>>> 
>>>> What do you think? Any issues we might have by doing this internally?
>>>> 
>>>> Cheers
>>>> Garren
>>> 
>> 
>> -- 
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>> 
> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/


Mime
View raw message