From dev-return-48580-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Thu May 16 07:39:01 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id F1D5518066B for ; Thu, 16 May 2019 09:39:00 +0200 (CEST) Received: (qmail 71957 invoked by uid 500); 16 May 2019 07:39:00 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 71939 invoked by uid 99); 16 May 2019 07:38:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 May 2019 07:38:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 352D81810E4 for ; Thu, 16 May 2019 07:38:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=medicmobile-org.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id aazsUkP7teoC for ; Thu, 16 May 2019 07:38:56 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 9692A6120A for ; Thu, 16 May 2019 07:38:56 +0000 (UTC) Received: by mail-wm1-f45.google.com with SMTP id i3so2449123wml.4 for ; Thu, 16 May 2019 00:38:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=medicmobile-org.20150623.gappssmtp.com; s=20150623; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=hrtfZGVhkbvM/C7CfVPYu/UludNDFcLm/Vqz/Tem4Vg=; b=nDt3VPXgFp5Oe/eWSc+tFjSNjz0TsTWe9mCy5LrJtVmHavHBERRZSQN+lALqpCCppb a3nat/nID+XsruaBM2+Zr6A9V5eAqXebiv/GCsIvNztNJKemR8IszpLxGoRnOCg9rR4e x8ozoby6v0V8z7xN5uuYVWhF5WOYVLxSJDvn81Nt4G3AureQnqfx39AgEN1XNGYKioQR m5zkXs0/DSWEWMGYwAIs2YfYj5fDxc1JE70uzwU+zW4k1COXbi0VElyH/QnlUE2IUzC8 OXeEGvM+6XyFR1tXGjxGDCtYJcWeznS3BQsUVdkl2i9dIOgqXPueDGqRkKB5hYBvhcre kKQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=hrtfZGVhkbvM/C7CfVPYu/UludNDFcLm/Vqz/Tem4Vg=; b=HI/lxBOeJDECi2O0LCehoNqLkNfWfXvTBEdrbwSzT0+a6gU5WIEcFnAaZNOD4ghoQ6 5FqPZ5SzMbT4/RfwMPvmmfPdC1MJrgLg4+Hy+0+widENxZ43YRYz7odALEjvcb8LXeVN rXKm37i+rREjcMHHn1EWC/Mwa3Blj2KtS5eTLBb3yx1ec55cLuwZGbwCUDy1gXgQX0y/ j+Fa3ObE5JE4B96rJ0JS8337QG0hkpfocc/qnbPW415+BMWvf/JWE4EkZ2jxBHG8kmE5 I8NfdVD07r9DfYkMzOkXgjgj/4td5Dr940zI8ElSH+6+0noI4V/8ArR3S3cGEph+qGPQ AaPA== X-Gm-Message-State: APjAAAUkyYkw1uledzs1ghkzIyxw1CYu+OW4BeotwH/SzJcLGZm/ryc2 gyFubBMaMFKysZb1ggyaZ73m54PWaes= X-Google-Smtp-Source: APXvYqw64xLhipay7+3KL1xnOXHRONbqtR0luEuCAG0NIu794bsMnWnAv47ShZ6pZh/G7gxGDljagQ== X-Received: by 2002:a1c:9c03:: with SMTP id f3mr27076364wme.67.1557992329811; Thu, 16 May 2019 00:38:49 -0700 (PDT) Received: from [192.168.1.204] ([2.24.105.7]) by smtp.gmail.com with ESMTPSA id z7sm3175344wme.26.2019.05.16.00.38.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 May 2019 00:38:48 -0700 (PDT) From: Stefan du Fresne Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: Design doc index switching Date: Thu, 16 May 2019 08:38:44 +0100 References: To: dev@couchdb.apache.org In-Reply-To: Message-Id: <00FEB1F0-E158-4403-AD95-778A7F2FAC49@medicmobile.org> X-Mailer: Apple Mail (2.3445.104.8) Hey Garren, Having this a native part of CouchDB seems like a really cool idea: we = have automated the manual dance you're talking about with our deployment = tooling, but it would be really nice not to have to! I'm not clear how it would work though, at least in terms of coherent = deployments. View changes are, like SQL migrations, an often = non-backwards compatible change that has to occur as your new code = deploys. Currently the naive approach is you deploy your new code alongside = design doc changes, which then block view queries on first request until = they're ready to go. The better approach is what you describe, which is what we do now, where = we extract our design documents out of our deployment bundle and place = them in a "staging" location to allow them to warm, then rename them and = do the actual code deployment once that's complete (managed by an = external deployment service we built). This importantly lets us split = the "warming" bit from the deployment bit: we only deploy new code once = the design documents that are shipped with that code is ready to go. How would you foresee this kind of flow happening here? Would there be a = way to query the design doc to know if it had flipped to the new version = yet? Would you be able to control when this flip occurs? Or would the = expectation be that your code handles both versions gracefully? As an example to mull over, let's say you have design doc v1, which has = view a. You push design doc v2, which has added view b, but has also = changed view a in some backwards incompatible way. While v2 is still = building and is not yet the active doc: - If you queried view a you'd get the v1 version, that's clear - If you queried view b you'd get... a 404? Some other custom code? - If you GET the design document what doc would you see? Presumably v2? - Could you query something to determine which version is currently = active? Or perhaps just whether there is a background version building = at all? Cheers, Stefan > On 16 May 2019, at 07:51, Garren Smith wrote: >=20 > Hi Everyone, >=20 > A common pattern we see for updating large indexes that can take a few = days > to build, is create a new design docs with the new updated views. Then = once > the new design doc is built, a user changes the new design doc=E2=80=99s= id to the > old design doc. That way the CouchDB url for the views remain the same = and > any requests to the design doc url automatically get the latest views = only > once they built. >=20 > This is an effective way of managing building large indexes, but the > process is quite complicated and often users get it wrong. I would = like to > propose that we move this process into CouchDB and let CouchDB handle = the > actual process. =46rom a users perspective, they would add a field to = the > options of a design document that lets CouchDB know, that this build = needs > to be built in the background and only replace the current index once = its > built: >=20 > ``` > { > "_id": "_design/design-doc-id", > "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2", > "views": { > "map-view": { > "map": "function (doc) {\n emit(doc._id, 1);\n}" > } > }, > "language": "javascript", > "options": { > "build_and_replace": true > } > } > ``` >=20 > I think this is something we could build quite effectively once we = have > CouchDB running on top of FoundationDB. I don=E2=80=99t want to = implement it for > version 1 of CouchDB on FDB, but it would be nice to keep this in mind = as > we build out the map/reduce indexes. >=20 > What do you think? Any issues we might have by doing this internally? >=20 > Cheers > Garren