From commits-return-35188-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Thu Nov 8 09:00:41 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A382618067A for ; Thu, 8 Nov 2018 09:00:40 +0100 (CET) Received: (qmail 87321 invoked by uid 500); 8 Nov 2018 08:00:39 -0000 Mailing-List: contact commits-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list commits@couchdb.apache.org Received: (qmail 87311 invoked by uid 99); 8 Nov 2018 08:00:39 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Nov 2018 08:00:39 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id A9F9A857F1; Thu, 8 Nov 2018 08:00:38 +0000 (UTC) Date: Thu, 08 Nov 2018 08:00:39 +0000 To: "commits@couchdb.apache.org" Subject: [couchdb-documentation] 01/01: Improve clustered purge documentation MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit From: jiangphcn@apache.org In-Reply-To: <154166403858.13588.12524372560134767046@gitbox.apache.org> References: <154166403858.13588.12524372560134767046@gitbox.apache.org> X-Git-Host: gitbox.apache.org X-Git-Repo: couchdb-documentation X-Git-Refname: refs/heads/COUCHDB-3326-clustered-purge-impr X-Git-Reftype: branch X-Git-Rev: f3f896362de26a09d040ea489725b7bc10891bb5 X-Git-NotificationType: diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated Message-Id: <20181108080038.A9F9A857F1@gitbox.apache.org> This is an automated email from the ASF dual-hosted git repository. jiangphcn pushed a commit to branch COUCHDB-3326-clustered-purge-impr in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git commit f3f896362de26a09d040ea489725b7bc10891bb5 Author: jiangph AuthorDate: Thu Nov 8 15:59:57 2018 +0800 Improve clustered purge documentation COUCHDB-3326 --- images/purge-checkpoint-docs.png | Bin 69408 -> 77925 bytes src/api/database/misc.rst | 5 +-- src/cluster/purging.rst | 92 ++++++++++++++++++++++----------------- 3 files changed, 53 insertions(+), 44 deletions(-) diff --git a/images/purge-checkpoint-docs.png b/images/purge-checkpoint-docs.png index 943db23..0480aa3 100644 Binary files a/images/purge-checkpoint-docs.png and b/images/purge-checkpoint-docs.png differ diff --git a/src/api/database/misc.rst b/src/api/database/misc.rst index 3cbfed7..7f5ef61 100644 --- a/src/api/database/misc.rst +++ b/src/api/database/misc.rst @@ -80,13 +80,12 @@ Server: CouchDB/2.0.0-2ccd4bf (Erlang OTP/18) { - "purge_seq": "3", + "purge_seq": null, "purged": { "c6114c65e295552ab1019e2b046b10e": { "purged": [ "3-c50a32451890a3f1c3e423334cc92745" - ], - "ok": true + ] } } } diff --git a/src/cluster/purging.rst b/src/cluster/purging.rst index 996ef86..76b1fe6 100644 --- a/src/cluster/purging.rst +++ b/src/cluster/purging.rst @@ -74,8 +74,8 @@ local checkpoint purge document: { "_id": "_local/purge-mrview-86cacdfbaf6968d4ebbc324dd3723fe7", "type": "mrview", - "purge_seq": "10", - "updated_on": "2018-05-09T08:41:37.183026Z", + "purge_seq": 10, + "updated_on": 1540541874, "ddoc_id": "_design/foo", "signature": "5d10247925f826ae3e00966ec24b7bf6" } @@ -93,6 +93,7 @@ Internal Replication ==================================== Purge requests are replayed across all nodes in an eventually consistent manner. Internal replication of purges consists of two steps: + 1. Pull replication. Internal replication first starts by pulling purges from target and applying them on source to make sure we don't reintroduce to target source's docs/revs that have been already purged on target. In this step, we use @@ -101,6 +102,7 @@ purge checkpoint documents stored on target to keep track of the last target's this ``purge_seq``, and replay them on source. This step is done by updating the target's checkpoint purge documents with the latest process ``purge_seq`` and timestamp. + 2. Push replication. Then internal replication proceeds as usual with an extra step inserted to push source's purge requests to target. In this step, we use local internal replication checkpoint documents, that are updated both on target @@ -129,45 +131,53 @@ consistency within the main database. Config Settings ==================================== -These settings ca be updated in the default.ini or local.ini: +These settings can be updated in the default.ini or local.ini: + ++-----------------------+--------------------------------------------+----------+ +| Field | Description | Default | ++=======================+============================================+==========+ +| max_document_id_number| Allowed maximum number of documents in one | 100 | +| | purge request | | ++-----------------------+--------------------------------------------+----------+ +| max_revisions_number | Allowed maximum number of accumulated | 1000 | +| | revisions in one purge request. | | ++-----------------------+--------------------------------------------+----------+ +| allowed_purge_seq_lag | Beside purged_infos_limit, allowed | 100 | +| | additional buffer to store purge requests | | ++-----------------------+--------------------------------------------+----------+ +| index_lag_warn_seconds| Allowed durations when index is not | | +| | updated for local purge checkpoint document| 86400 | ++-----------------------+--------------------------------------------+----------+ + + +During a database compaction, we check all checkpoint purge docs. A client (an +index or internal replication job) is allowed to have the last reported +``purge_seq`` to be smaller than the current database shard's ``purge_seq`` by +the value of ``(purged_infos_limit + allowed_purge_seq_lag)``. If the client's +``purge_seq`` is even smaller, and the client has not checkpointed within +``index_lag_warn_seconds``, it prevents compaction of purge trees and we have to +issue the following log warning for this client: + +.. code-block:: text + + Purge checkpoint '_local/purge-mrview-9152d15c12011288629bcffba7693fd4’ + not updated in 86400 seconds in + <<"shards/00000000-1fffffff/testdb12.1491979089">> + +If this type of log warnings occurs, check the client to see why the processing +of purge requests is stalled in it. + +There is a mapping relationship between design document of indexes and local +checkpoint docs. If a design document of indexes is updated or deleted, the +corresponding local checkpoint document should be also automatically deleted. +But in an unexpected case, when a design doc was updated/deleted, but its +checkpoint document still exist in a database, the following warning will be +issued: .. code-block:: text - [purge] - max_document_id_number = 100 - max_revisions_number = 1000 - allowed_purge_seq_lag = 100 - index_lag_warn_seconds = 86400 - - 1. ``max_document_id_number`` (default 100) - Allowed maximum number of documents in one purge request - 2. ``max_revisions_number`` (default 1000) - Allowed maximum number of accumulated revisions in one purge request. - 3. ``index_lag_warn_seconds`` (default 86400 sec or 1 day) - During a database compaction, we check all checkpoint purge docs. A client (an - index or internal replication job) is allowed to have the last reported - ``purge_seq`` to be smaller than the current database shard's ``purge_seq`` by - the value of ``(purged_infos_limit + index_lag_warn_seconds)``. If the client's - ``purge_seq`` is even smaller, and the client has not checkpointed within - ``index_lag_warn_seconds``, it prevents compaction of purge trees and we have to - issue the following log warning for this client: - - :: - Purge checkpoint '<<"_design/bar">>' not updated in 86400 seconds - - If this type of log warnings occurs, check the client to see why the processing - of purge requests is stalled in it. - - There is a mapping relationship between design document of indexes and local - checkpoint docs. If a design document of indexes is updated or deleted, the - corresponding local checkpoint document should be also automatically deleted. - But in an unexpected case, when a design doc was updated/deleted, but its - checkpoint document still exist in a database, the following warning will be - issued: - - :: - "Invalid purge doc '<<"_design/bar">>' on database - <<"shards/00000000-1fffffff/testdb12.1491979089">> - with purge_seq '50'" - - If this type of log warnings occur, remove the local purge doc from a database. + "Invalid purge doc '<<"_design/bar">>' on database + <<"shards/00000000-1fffffff/testdb12.1491979089">> + with purge_seq '50'" + +If this type of log warnings occur, remove the local purge doc from a database.