From dev-return-48583-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Thu May 16 21:03:45 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8FDA118066B for ; Thu, 16 May 2019 23:03:45 +0200 (CEST) Received: (qmail 98685 invoked by uid 500); 16 May 2019 21:03:44 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 98670 invoked by uid 99); 16 May 2019 21:03:44 -0000 Received: from Unknown (HELO mailrelay2-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 May 2019 21:03:44 +0000 Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com [66.111.4.227]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id 5DCF62C75 for ; Thu, 16 May 2019 21:03:44 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailauth.nyi.internal (Postfix) with ESMTP id 8B9E724737 for ; Thu, 16 May 2019 17:03:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 16 May 2019 17:03:43 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddttddgudehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephfgtgfgguffffhfvjgfkofesth hqmhdthhdtjeenucfhrhhomheptfhosggvrhhtucfurghmuhgvlhcupfgvfihsohhnuceo rhhnvgifshhonhesrghprggthhgvrdhorhhgqeenucffohhmrghinhepnhgvihhghhgsoh hurhhhohhougdrihgvnecukfhppedukeehrddvvddvrddvjedrvdegudenucfrrghrrghm pehmrghilhhfrhhomheprhhnvgifshhonhdomhgvshhmthhprghuthhhphgvrhhsohhnrg hlihhthidqleefgedvtddvjedvqdduudelgeejtdejjedqrhhnvgifshhonheppegrphgr tghhvgdrohhrghesfhgrshhtmhgrihhlrdhfmhenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from [198.18.12.82] (unknown [185.222.27.241]) by mail.messagingengine.com (Postfix) with ESMTPA id E23EA80062 for ; Thu, 16 May 2019 17:03:42 -0400 (EDT) From: Robert Samuel Newson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: Design doc index switching Date: Thu, 16 May 2019 22:03:41 +0100 References: <00FEB1F0-E158-4403-AD95-778A7F2FAC49@medicmobile.org> To: CouchDB Developers In-Reply-To: Message-Id: <036BDF3D-2550-47D9-850A-7782661EF480@apache.org> X-Mailer: Apple Mail (2.3445.104.8) I suggest an alternative; the new design document could include the _id = of design document it=E2=80=99s replacing = (=E2=80=9C_replaces=E2=80=9D:=E2=80=9D_design/foo=E2=80=9D). On = completion of the view build of the new design document, CouchDB itself = updates the named _id to the same content as the new design document = (strictly, only the parts needed to make the view sig match) (perhaps it = also deletes the new document). The advantage to this is that queries to the original design document = continue to work throughout and at no point is there a discrepancy = between the design documents contents (the map and reduce functions, = etc) and the results you get from it. B. > On 16 May 2019, at 14:55, Jan Lehnardt wrote: >=20 > +1 on solving this for all users, and same caveats as Stefan raises :) >=20 >> On 16. May 2019, at 09:38, Stefan du Fresne = wrote: >>=20 >> Hey Garren, >>=20 >> Having this a native part of CouchDB seems like a really cool idea: = we have automated the manual dance you're talking about with our = deployment tooling, but it would be really nice not to have to! >>=20 >> I'm not clear how it would work though, at least in terms of coherent = deployments. View changes are, like SQL migrations, an often = non-backwards compatible change that has to occur as your new code = deploys. >>=20 >> Currently the naive approach is you deploy your new code alongside = design doc changes, which then block view queries on first request until = they're ready to go. >>=20 >> The better approach is what you describe, which is what we do now, = where we extract our design documents out of our deployment bundle and = place them in a "staging" location to allow them to warm, then rename = them and do the actual code deployment once that's complete (managed by = an external deployment service we built). This importantly lets us split = the "warming" bit from the deployment bit: we only deploy new code once = the design documents that are shipped with that code is ready to go. >>=20 >> How would you foresee this kind of flow happening here? Would there = be a way to query the design doc to know if it had flipped to the new = version yet? Would you be able to control when this flip occurs? Or = would the expectation be that your code handles both versions = gracefully? >>=20 >> As an example to mull over, let's say you have design doc v1, which = has view a. You push design doc v2, which has added view b, but has also = changed view a in some backwards incompatible way. While v2 is still = building and is not yet the active doc: >> - If you queried view a you'd get the v1 version, that's clear >> - If you queried view b you'd get... a 404? Some other custom code? >> - If you GET the design document what doc would you see? Presumably = v2? >> - Could you query something to determine which version is currently = active? Or perhaps just whether there is a background version building = at all? >>=20 >> Cheers, >> Stefan >>=20 >>> On 16 May 2019, at 07:51, Garren Smith wrote: >>>=20 >>> Hi Everyone, >>>=20 >>> A common pattern we see for updating large indexes that can take a = few days >>> to build, is create a new design docs with the new updated views. = Then once >>> the new design doc is built, a user changes the new design doc=E2=80=99= s id to the >>> old design doc. That way the CouchDB url for the views remain the = same and >>> any requests to the design doc url automatically get the latest = views only >>> once they built. >>>=20 >>> This is an effective way of managing building large indexes, but the >>> process is quite complicated and often users get it wrong. I would = like to >>> propose that we move this process into CouchDB and let CouchDB = handle the >>> actual process. =46rom a users perspective, they would add a field = to the >>> options of a design document that lets CouchDB know, that this build = needs >>> to be built in the background and only replace the current index = once its >>> built: >>>=20 >>> ``` >>> { >>> "_id": "_design/design-doc-id", >>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2", >>> "views": { >>> "map-view": { >>> "map": "function (doc) {\n emit(doc._id, 1);\n}" >>> } >>> }, >>> "language": "javascript", >>> "options": { >>> "build_and_replace": true >>> } >>> } >>> ``` >>>=20 >>> I think this is something we could build quite effectively once we = have >>> CouchDB running on top of FoundationDB. I don=E2=80=99t want to = implement it for >>> version 1 of CouchDB on FDB, but it would be nice to keep this in = mind as >>> we build out the map/reduce indexes. >>>=20 >>> What do you think? Any issues we might have by doing this = internally? >>>=20 >>> Cheers >>> Garren >>=20 >=20 > --=20 > Professional Support for Apache CouchDB: > https://neighbourhood.ie/couchdb-support/ >=20