From user-return-31649-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Tue Aug 28 15:12:45 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E8B37180621 for ; Tue, 28 Aug 2018 15:12:44 +0200 (CEST) Received: (qmail 70598 invoked by uid 500); 28 Aug 2018 13:12:43 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 70586 invoked by uid 99); 28 Aug 2018 13:12:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Aug 2018 13:12:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B4855C06AD for ; Tue, 28 Aug 2018 13:12:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.012 X-Spam-Level: X-Spam-Status: No, score=-0.012 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=zyn.onmicrosoft.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id d7fIvZbjhJd6 for ; Tue, 28 Aug 2018 13:12:39 +0000 (UTC) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00050.outbound.protection.outlook.com [40.107.0.50]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 853635F3B8 for ; Tue, 28 Aug 2018 13:12:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyn.onmicrosoft.com; s=selector1-zynstra-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U9WEFoNbpWwGYX5ShTAgu8A36DLY0nKJXzFwb1QvQXI=; b=Jvnvjn6tDJ62x3curnIGRO/dsbIX3N4dUHntjU3s50M+8Ya2zZXgSv5yWHvRICZaKuCCHZrhlHi4SmSOjzuYlHnROTMNqfHCHlVxp43/WACcND4XaAmGQkweSdcAjuD6w61XynQyan5zf6AB/oLiHDSju79GEJYNccWbt5BKXhA= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=james.dingwall@zynstra.com; Received: from [192.168.10.161] (193.63.64.25) by AM6P192MB0343.EURP192.PROD.OUTLOOK.COM (2603:10a6:209:3a::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1080.13; Tue, 28 Aug 2018 13:12:29 +0000 Subject: Re: Dynamic Filtered Replication To: user@couchdb.apache.org References: From: James Dingwall Message-ID: Date: Tue, 28 Aug 2018 14:12:26 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Originating-IP: [193.63.64.25] X-ClientProxiedBy: DB3PR0202CA0023.eurprd02.prod.outlook.com (2603:10a6:8:1::36) To AM6P192MB0343.EURP192.PROD.OUTLOOK.COM (2603:10a6:209:3a::33) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9191d33c-060b-400c-4c80-08d60ce7e835 X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(2017052603328)(7153060)(7193020);SRVR:AM6P192MB0343; X-Microsoft-Exchange-Diagnostics: 1;AM6P192MB0343;3:XVJDsvIWF/fscpCAiV5sKLmyKWM8O8FZzhdqEpWqwMpMHEDQwmv6UgdsfjYdChEDXR9YNHrycqM26CHRLZTEGX3hR2NaJ4H9URfA3y4ByQoZLvq52jN7g3VAzfCNbXPurRbgf6zieX0T5/RqcXu4unnkkSM0q6xnruUSCOpki4y96sZx6pW+GbJEQD9z97Rj0cCymmC4kDkIkBkSwHI7tOuga+wjiNzz+AGWtl1JWfrEgXTq45nediTioB449TSS;25:329/AWP/LLXWIzQyozC1f72/QAyC60KyXDKLRh+JRskQEVVjXi2H8z0lT8zyJOW2g3GOrabFHr1o4fcRVcrrKHtFlQdu2mZun46BIUH/MJlqFykvPjVvd7NcPYgGSsA5v4DvQgIg2IcHJvBJ+Xs0S6Evj3X4GQc/gAL5JHsauthAqGay5Rr6+8jRJZF1mSCMM61QoLBO5q0k0P93FKhEitvsEwNTVNa9qWCRGdDfOxWzhJtvPyb6eKChGeaGeMa7vvUUIJQN2KzthurBLQa6Ap2SDZMIvQ3/0elEUCS7/TktZo9pTF0WUjiVgLgomQooEq7/KUhR9HloBqyzbx0R/Q==;31:MzqwYMO6pZgSW9GrmD4J/KbQpvJky049ick41kSRSfAnyL68dT456uWzg76KFFXDVI3zw0ngeuG9wUd+yicKUR9BJO43ZV37smI12WO8NjfQniCq6UV0IAq/+yhI1sAqAp+IDffbTiLisUXKYRHmZIlInjAuA69kcDLbt+sXmcoEk17T6iwGOmCyylXfEjx7MdK+n51v3R9LEc6dITO7g4wjo9JIuoD4XbXKwsEBoE8= X-MS-TrafficTypeDiagnostic: AM6P192MB0343: X-Microsoft-Exchange-Diagnostics: 1;AM6P192MB0343;20:W/seESRDn27oiwXbiuJMGg4pgqSGs980Lldlfw1FttVH6jJXvQCoxd9p5+FrJEXN3i3zIX7Yg4w/UMkiayw+TZ0WYd7STNc+tna9EVR/E9hULkplBUnbpWHxkHqGyHV8GzOhMDwo4gfwFVhW0eWcnIqWPYJxxeWrsAw1kUK51KmCx2rINX30QEoPLbvmeMSXMgv+Jus9DpHZlOUtcwtDweK2+T6KnyUt9mMVjVeXOyCXgZh+ScjVclk03Y5FHkwO;4:r4y/bvyiq/o9PKgoP4x8qFxQlL2gW/9oH7nLtfclu+kmw/U3WPhSYSxxzfgzk425/4x5iq/a/O8dpV6SBmE/PsdJ+xEsM/ruq/LnGN2v4ka1KHNAJ+I2wvIO+XzLL9PxGqXgwaJH+GTG8+OnodHjDf2D17HMaoskNYYUaW5vxmH8vil6d/DazDi9VG6aJHFtkZ8Lj3hvR+CalsnJXVP3ydvSAH2qs5pWQmT3qetWiRwgoHpwtlYq/UCvTmyaBlWAlo5YmBm0FZXCvy5OLdJIfA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(8121501046)(5005006)(10201501046)(3231311)(944501410)(52105095)(3002001)(93006095)(93001095)(149027)(150027)(6041310)(20161123558120)(20161123564045)(20161123562045)(201703131423095)(201703011903075)(201702281528075)(20161123555045)(201703061421075)(20161123560045)(201708071742011)(7699016);SRVR:AM6P192MB0343;BCL:0;PCL:0;RULEID:;SRVR:AM6P192MB0343; X-Forefront-PRVS: 077884B8B5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6049001)(7966004)(396003)(366004)(39840400004)(346002)(376002)(136003)(53754006)(199004)(189003)(52116002)(2486003)(23676004)(31696002)(8676002)(36756003)(97736004)(76176011)(86362001)(31686004)(65806001)(66066001)(53546011)(2361001)(65956001)(67846002)(64126003)(476003)(2616005)(956004)(11346002)(446003)(77096007)(386003)(106356001)(47776003)(6116002)(3846002)(105586002)(486006)(52146003)(26005)(16526019)(186003)(58126008)(305945005)(2906002)(7736002)(81166006)(16576012)(316002)(2351001)(117156002)(5660300001)(3480700004)(50466002)(508600001)(6246003)(25786009)(7116003)(6916009)(6666003)(65826007)(14444005)(6486002)(5024004)(44832011)(68736007)(8746002)(53936002)(8936002)(229853002)(81156014)(440614002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6P192MB0343;H:[192.168.10.161];FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; Received-SPF: None (protection.outlook.com: zynstra.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtBTTZQMTkyTUIwMzQzOzIzOnROSFVZeUhJZGhxcmZVcW9OY05LMDdsL2l1?= =?utf-8?B?ZmdSWkF6dEJ6aEh4dDdrcm1tYlBzQUtXZGdKcDM2RWVkb3dZV3FTcklONVRo?= =?utf-8?B?S3lQWVplaW90VzU5TUkzWTBjVitxRFZ0d0JtQmtMN1hlMVNVQ3ozQTdBa3Y0?= =?utf-8?B?aldndFhUNkdRTDQwVTA3eVhUWW1oVEdibkJJRU9rVDhTRGtRektITGIvb3VC?= =?utf-8?B?RjlMa0Y5Y0E2TGJGUzZ3TXp3eWR4a3FldXlDMEVZTkFKMEJzMEhKQ2J4UVRs?= =?utf-8?B?SFVRRjlxcVByRW1wd2JPaGRpZ0V3YVNKT1VObEhaTWRWT3YwWFpkZzdtUEFG?= =?utf-8?B?QityYlQwZHVLd08zVklJcHVmZEpxUjhEdjFBTXRWN3NMUUw2WXZrTkpKY3lM?= =?utf-8?B?NEdteEw5S2w4dUFZQlBtTS9CUGJ4V3BpTUtLNmg2VUdZYUpLMW50WXdNcXlJ?= =?utf-8?B?TVlNcXo2dzE3MnJjeHBZb0c2OXJDY0dmalp6Yi85OFlRSW9MM0c0OVZrOFIw?= =?utf-8?B?RlFoZ3VXaTRUMVVsdDIrY0xidElJbGhWVXFnNzIraWtjYU5xZjR1MnB4TXFL?= =?utf-8?B?TFdSU2lOditCa1J2VmxKclVkTzdYOWZ1VE80L01wdUFYR25sT3BJL3plTW9U?= =?utf-8?B?ZEhXQTczME1tRFdKNjk0dTZTMHlrT21ncTgvcTl5c1RtUHZwR0NRTWN6SmJ0?= =?utf-8?B?alozNnNDNW1TeERlK1REUFNQcHJqY3ZjQTVQZFlmZHkzUlNoN1V4UUY1T1V3?= =?utf-8?B?NUo4ZHFIdWtnZkdNZk9ld0E0N2d1U0VRZHZObnZIMG90Vkllb0tvcVVVemFF?= =?utf-8?B?SkhDVzJnOTQ5RjhOYlU2SDFYWEZZK3ozTW5zZkZISUNydDdmMTdvL0xrcEZ4?= =?utf-8?B?N1NWdzVrNmZwQndKaGc5ZFdTTW1Hd2FtMG1STWcxSjkzVXQzSXVYK3UwRXda?= =?utf-8?B?cHIyLzBENS9LUTVnZmUzMk1UeThDT2llWTZmczEzdUVGMWplVjViZDdYU3NP?= =?utf-8?B?THZFSDQ1azNiSmtZdGRNUGdBY3JoOHRiTHlEak1HMm1uSDRBS01QUm9KVEJw?= =?utf-8?B?UkRYQmZXejc5RGJ4bGZtT25zWnA4cU92VHJTMVIrbzMxeStIZlFZNndHY0I5?= =?utf-8?B?dU5WUTVZbVBkeVJjV09DRGVYanlXblRkOTJHWGdLdnVVV1I2YUcwVkk4OE1S?= =?utf-8?B?aWUrdDIxd2Voc1lzV2wzbUZwS0pKd3FpUXlvWXRmSXQ3aWdZVlFkbVFVakls?= =?utf-8?B?YXNaWDJUcUt6YUlhSjJ0bmZMbi85WUNoTGdOU2RPeVpseWZHTWZub2U1cDg4?= =?utf-8?B?cE1DZGE2RU94ZkxhbUR5SVlqdlpYbXg0YlBJMjVYSlpsbGJnQ1BCNzU2eEwx?= =?utf-8?B?dFJmM1g5NFhkcnJrN3Azd1UvNGw5YU5ZZjRmRjkvcXZXTi9YVFNLcVpia3N4?= =?utf-8?B?NFNNL21jVFZzU0tqNFpVN1VGbUZlR245MDg0VExLK1dEUEpITTd6NEx1MVgv?= =?utf-8?B?KzZDclpoTUh1U3c4RTd4M29PMkpUZFFsTEZweXFVYll6Nk1FOThVOUxXTGJy?= =?utf-8?B?bG8rZnk3WWtLeUc0cDM0QXMxdmcrTkd6M1JQbXc1cklac3VnWEZSdW9rOVpG?= =?utf-8?B?VGlVL1ZEZEd0NUtJL25weVZqQjNWbVRnUFNYYjdqdWZiYWY3NkQwU0pMNUdX?= =?utf-8?B?eTBmTk9iMnZSZG92THZjZHZsczdFWmtZSVBKWmxTM25YRndtZ3pjV0tPZGRh?= =?utf-8?B?dDM1RlJwd2g0SXBGZ0FNZDJ4NFdncm5lTTZyMGlETlZBL3ozODJ5eEI4T3Q1?= =?utf-8?B?SkYvUkpWWHdZT0x1aUlUOG9JcnVSTGx6TTVnRllnak5tVDVDOVREQUdXb0Q4?= =?utf-8?B?R3dWNnp4NTFidUN2TWJOa01pSUhrQmUzNk1SbUdVV2pjSVVNL094NGU4cjlT?= =?utf-8?B?SWEvMFQxcWhXQlFIM3FMNnVyUk03aEFTK1p0UmplWEdlcDB6WFpPYThmQ0Rm?= =?utf-8?B?UGZtdnVocDNHUWJCNGZCa0N5R3NjTmRrUGtqV2o4NDZJY0ZkeWFxdDVEb250?= =?utf-8?B?Yy9qeHV3bVBHOVlocjd3b003UmJKOUoxL29adGhtN29lUmhEQ2NvVURCc01V?= =?utf-8?B?azVNckV0dHQzMHEvWXhFSmVtZnF1dU9jcVB6ZjVjU0NqSXhrSVNHbWZkd1dE?= =?utf-8?Q?w1UaWGznwz2S+CvUrtnHw3TT1ocHlEjpRRnnEhijkQ=3D?= X-Microsoft-Antispam-Message-Info: R0A6AhNw/47SFy35ehKErtW/AkmiC94iUwIBdyC8YxzPIrrsPiLC2I1kZc2yZJQzVXrLHC/UDhmvOpbdYnXiBSkf2yL2dvAsxxoHswTjfV9NBePYvGuyrzhwUxvgS489JV+PHoG6PcWQAvBNojCZZL+/tig5I+3ZMAWrLiocE5gAm4HY1DQSKGjcE61qAZ4qd+5hxJfbCPCybfVmC+Zxn7EBOBsiAKgeJyVTZsF/zZTmg0RjKGSZtE3pnTLZZMhYOkNcuoTP1PTMxxuWp/gVMZMqAjFEkaA/v5EyXe3Awk0DgGC9fdWlHv501ULUxWeGF+L2HAO5rqcC0NPjKAowXJA8bVtYYXV4fJdsW63K7BE= X-Microsoft-Exchange-Diagnostics: 1;AM6P192MB0343;6:Vi2DWqSW3zlWvhStNJBKoQCDOwUPyH6KYWjDPfSR8Bp6SLlT5fFYhMRBONhgX0xfGs/juvtplQzHiuDxmxbvHBEBNUESSiMAvMV9RIXf7iOx54+Rn8gLrgbqKxigEgFroZDnkOhNGbGe7UGvJH1Qq6EOvuhDsaBq5HoAw4RtxZaAObKr6UHZ0/pUorjazgKFHOjEmM4TbDch+kLIPwsty076FelSWwA7vXGjvDe882BAgKo8rJ8bE0Yr2t6N+eUQSfHQMJwW2hrDQLXi2gx1Ne+1UvoYHJLztn4bF9/f5nruFC6cEGb7xp0dX/ouuQhkbniFx8bRyKtm8WehHND6Dx61KaKQUiTGuaVtyPLHpDCx9FYrbX3awBzq5iWyfmfFhpRPyEkImx3BJpRsq+kke7ohyhl+VIQwyczzYxBOBniOpWVimcFO52WOl+aag+ZcNT1/qQ12werdC8mBp4qkjA==;5:BF+ZHjiqtCbSHpS6gV90EGG8U1zx03eLDHI7weDIP58ZpjOQ0VCtKrIaBfub35SUldU5SJQhAsxLEMCpE95sZleAh6I1LqXCttgkK74d75i3phIfUBeiPLfUL/me/kkN19iC1aG6JH1G3dY33liR+3gH4KaFT4CjhLr2eWk6ue8=;7:U3nAZib5Pyy8wq3chCo/AK5eCDPHhpnQI+4QGC0K21kJJPhvSYI03VzLk1buN+5IpuxfmFseIcIJzCSHohECOm0xGWpezMSrL1y/uM86Cc3bJdkBAmBbb7gmjENFUSDZ7c9QlY/NVff3toU20+BllVUlZkEyvrxcnRK+bGvKLBLm2MvM4s3IT9CWpTAUeq5FaUl0HRjrWKwSsEgZFLgCScerSMxKVaLLSHMyTf7fVKZ+mBlQiOL+HOF8cqCP9Iip SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: zynstra.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Aug 2018 13:12:29.1195 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9191d33c-060b-400c-4c80-08d60ce7e835 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4c2b4c7a-32de-4e00-8ebe-361d80e7a665 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6P192MB0343 On 28/08/18 13:51, Andrea Brancatelli wrote: > Hello everybody. > > I have a pretty academic question about CouchDB replication=E2=80=A6 > > I=E2=80=99m working on creating a setup to test my idea but I thought ask= ing could save me some headaches. > > This is the scenario. > > Let=E2=80=99s suppose I have two database: Master and Slave with a Filter= ed Replication that uses some information stored in the _user database to d= etermine wether to replicate the document or not =E2=80=A6 > > Now we put 100 docs in the Master DB, so Master Update SEQ reaches, let= =E2=80=99s say, 100. > > The filter has filtered the replicated documents and on the Slave DB we h= ave, let=E2=80=99s say 40 documents. > > The condition in the _user database changes and the new condition would m= ach 60 documents of the Master. > > Is there an =E2=80=9Ceasy=E2=80=9D way to refresh the sincronization betw= een the two database and have the 20 missing docs synced? > > Maybe resetting the replication would automagically fix this (like the re= plication skipping the docs that are already in the Slave db?) > > How would you handle such a situation? Especially when the amount of docs= is not 100... :-) My experience with 1.x are that this can be done by: 1. recreate the replication with a new _id 2. recreate the replication with the same _id but add some query parameter to make it unique from the previous 3. recreate the replication with the same _id but remove the _local document relating to the replication from master and slave dbs. (Use the document name is _local/ where replication_id can be found from the related task and is processed with (python) task["replication_id"].split("+")[0]) 4. if there is no unique content in the slave and you can afford to miss it, just delete it and it will be re-created if your replication is allowed to create_target. By observation the status is maintained in a _local document using a checksum of the replication JSON (perhaps the same calculation used to generate the _rev) so unless you change the JSON defining the replication it will resume from the last sequence recorded in the _local document. My usual approach is 3 although poking at the _local document probably isn't supported. The sequence I use is: - delete replication definition (_deleted: true, not an HTTP DELETE otherwise replication processes may not be correctly terminated) - remove _local documents from master and slave (which may not be present depending on the status of the existing replication) - re-create the replication with the same JSON content as before The main issue is when your condition for replication changes such that documents already present in the slave would no longer be replicated according to your new criteria, in this case 4 is the only solution. > > I hope my question is clear enough. > > Thanks a lot. > > ------- > Andrea Brancatelli > > James Zynstra is a private limited company registered in England and Wales (regis= tered number 07864369). Our registered office and Headquarters are at The I= nnovation Centre, Broad Quay, Bath, BA1 1UD. This email, its contents and a= ny attachments are confidential. If you have received this message in error= please delete it from your system and advise the sender immediately.