From dev-return-49260-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Thu Apr 23 16:30:20 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id BC7A2180608 for ; Thu, 23 Apr 2020 18:30:19 +0200 (CEST) Received: (qmail 84978 invoked by uid 500); 23 Apr 2020 16:30:19 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 84961 invoked by uid 99); 23 Apr 2020 16:30:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Apr 2020 16:30:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 73B98C05D2 for ; Thu, 23 Apr 2020 16:30:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.401 X-Spam-Level: X-Spam-Status: No, score=-0.401 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=rsn.io header.b=QsVR/StY; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=1ttd+kI7 Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id YgZxrZGCgA9s for ; Thu, 23 Apr 2020 16:30:15 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=64.147.123.21; helo=wout5-smtp.messagingengine.com; envelope-from=bob@rsn.io; receiver= Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id C1403BB8FE for ; Thu, 23 Apr 2020 16:30:14 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 83A7948E for ; Thu, 23 Apr 2020 12:30:07 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Thu, 23 Apr 2020 12:30:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rsn.io; h= content-type:content-transfer-encoding:from:mime-version:subject :date:message-id:references:in-reply-to:to; s=fm2; bh=t848sGn3yl y/4HgHBxkH/NBjLvOUptQCb61sjdU+Pnw=; b=QsVR/StYpXIDSi5Cl6ctGeOMYG TluOh8WtBgZtZWz5Xs6x4QrIZVetkz5IWEsKOKFUG4DLJFZUkF/KcC4Pgbi1gBox 3WqPyEs4iTPahMUGfx2CPH/tLd55LO8WuZBoqD8cbj+d9QRGKtqg85dZH3jlB0PG HI487SJO+o3AscSd4AZ1bj7bZfqcpeCTQl4RXa7+iKb/REtDVjc7gSdAMYd8tyK6 sAJcbiUl3DsUAhUyhDHvxJ0mu+LvjR5fMjJf/o89+9pNKSoK4TOexKz0YVhYqaks Mvvx14rCZ9B9BeMdpMDlFmGNaRsmuxFhIgBj8Of07PI4wng4rfL+tHVPM9ZA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=t848sGn3yly/4HgHBxkH/NBjLvOUptQCb61sjdU+P nw=; b=1ttd+kI7Z6LgEj4ZVAavlVpnVP169Wjx4Lr6fjcGVIhANEgmLhMVhhkyQ V700eRHL9pgnvU8GsjT4RFDbiFyJUGFr4tQSSy4hyhsdRBzznmuw6MiNSW6xqOn/ SPcvvyFLUvnHYRm8PPMRZXQ3mqkpuIaJ6KAUz6Cz53Glw6nSV0vuBskLCHYsOMHK RSaXJrzWSUm4eDcBKiaDv44NFgDFsyTPEvhzZC1QR/AM7/fEprp/f0yJf3mzkec8 LmI3kXtVBZimpOdCfRv/FBkGWwr5XgBFz43W2cUoyo3oIEafLXn6PHBkRnGy7ZSn es1HL75H+/146Ic5tDAubfgsYbpWA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrgeelgdejkecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecuogfuuhhsphgvtghtffhomhgrihhnucdlgeelmdenuc fjughrpegtgffhggfufffkfhgjvffosehtqhhmtdhhtdejnecuhfhrohhmpeftohgsvghr thcupfgvfihsohhnuceosghosgesrhhsnhdrihhoqeenucffohhmrghinhepghhithhhuh gsrdgtohhmpdhgihhthhhusgdrihhonecukfhppeekledrvdefkedrudehgedrudejjeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsohgsse hrshhnrdhioh X-ME-Proxy: Received: from [10.0.3.11] (unknown [89.238.154.177]) by mail.messagingengine.com (Postfix) with ESMTPA id 7A13A3065D40 for ; Thu, 23 Apr 2020 12:30:06 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Robert Newson Mime-Version: 1.0 (1.0) Subject: Re: [DISCUSS] Streaming API in CouchDB 4.0 Date: Thu, 23 Apr 2020 17:30:05 +0100 Message-Id: References: In-Reply-To: To: dev@couchdb.apache.org X-Mailer: iPhone Mail (17D50) cursor has established meaning in other databases and ours would not be very= close to them. I don=E2=80=99t think it=E2=80=99s a good idea.=20 B.=20 > On 23 Apr 2020, at 11:50, Ilya Khlopotov wrote: >=20 > =EF=BB=BF >>=20 >> The best I could come up with is replacing page with >> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs > Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor). >=20 >> On 2020/04/23 08:54:36, Garren Smith wrote: >> I agree with Bob that page doesn't make sense as an endpoint. I'm also >> rubbish with naming. The best I could come up with is replacing page with= >> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs >> All the fields in the bookmark make sense except timestamp. Why would it >> matter if the timestamp is old? What happens if a node's time is an hour >> behind another node? >>=20 >>=20 >>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov wrote= : >>>=20 >>> - page is to provide some notion of progress for user >>> - timestamp - I was thinking that we should drop requests if user would >>> try to pass bookmark created an hour ago. >>>=20 >>> On 2020/04/22 21:58:40, Robert Samuel Newson wrote:= >>>> "page" and "page number" are odd to me as these don't exist as concepts= , >>> I'd rather not invent them. I note there's no mention of page size, whic= h >>> makes "page number" very vague. >>>>=20 >>>> What is "timestamp" in the bookmark and what effect does it have when >>> the bookmark is passed back in? >>>>=20 >>>> I guess, why does the bookmark include so much extraneous data? Items >>> that are not needed to find the fdb key to begin the next response from.= >>>>=20 >>>>=20 >>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov wrote: >>>>>=20 >>>>> Hello everyone, >>>>>=20 >>>>> Based on the discussions on the thread I would like to propose a >>> number of first steps: >>>>> 1) introduce new endpoints >>>>> - {db}/_all_docs/page >>>>> - {db}/_all_docs/queries/page >>>>> - _all_dbs/page >>>>> - _dbs_info/page >>>>> - {db}/_design/{ddoc}/_view/{view}/page >>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page >>>>> - {db}/_find/page >>>>>=20 >>>>> These new endpoints would act as follows: >>>>> - don't use delayed responses >>>>> - return object with following structure >>>>> ``` >>>>> { >>>>> "total": Total, >>>>> "bookmark": base64 encoded opaque value, >>>>> "completed": true | false, >>>>> "update_seq": when available, >>>>> "page": current page number, >>>>> "items": [ >>>>> ] >>>>> } >>>>> ``` >>>>> - the bookmark would include following data (base64 or protobuff???): >>>>> - direction >>>>> - page >>>>> - descending >>>>> - endkey >>>>> - endkey_docid >>>>> - inclusive_end >>>>> - startkey >>>>> - startkey_docid >>>>> - last_key >>>>> - update_seq >>>>> - timestamp >>>>> ``` >>>>>=20 >>>>> 2) Implement per-endpoint configurable max limits >>>>> ``` >>>>> _all_docs =3D 5000 >>>>> _all_docs/queries =3D 5000 >>>>> _all_dbs =3D 5000 >>>>> _dbs_info =3D 5000 >>>>> _view =3D 2500 >>>>> _view/queries =3D 2500 >>>>> _find =3D 2500 >>>>> ``` >>>>>=20 >>>>> Latter (after few years) CouchDB would deprecate and remove old >>> endpoints. >>>>>=20 >>>>> Best regards, >>>>> iilyak >>>>>=20 >>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc wrote: >>>>>> Hello everyone, >>>>>>=20 >>>>>> I'd like to discuss the shape and behavior of streaming APIs for >>> CouchDB 4.x >>>>>>=20 >>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets >>>>>> read from the database. These are the endpoints I was thinking of: >>>>>>=20 >>>>>> _all_docs, _all_dbs, _dbs_info and query results >>>>>>=20 >>>>>> I want to focus on what happens when FoundationDB transactions >>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1= ] >>>>>> feeds, will crash or freeze. The reason is because the >>>>>> transaction_too_old error at the end of 5 seconds is retry-able by >>>>>> default, so the request handlers run again and end up shoving the >>>>>> whole request down the socket again, headers and all, which is >>>>>> obviously broken and not what we want. >>>>>>=20 >>>>>> There are few alternatives discussed in couchdb-dev channel. I'll >>>>>> present some behaviors but feel free to add more. Some ideas might >>>>>> have been discounted on the IRC discussion already but I'll present >>>>>> them anyway in case is sparks further conversation: >>>>>>=20 >>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue= >>>>>> streaming the data from the next key after last emitted in the >>>>>> previous transaction. Document the API behavior change that it may >>>>>> present a view of the data is never a point-in-time[4] snapshot of th= e >>>>>> DB. >>>>>>=20 >>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries >>>>>> don't have to change to continue using these CouchDB 4.0 endpoints >>>>>> - This is the easiest to implement since it would re-use the >>>>>> implementation for _changes feed (an extra option passed to the fold >>>>>> function). >>>>>> - Breaks API behavior if users relied on having a point-in-time[4] >>>>>> snapshot view of the data. >>>>>>=20 >>>>>> B) Simply end the stream. Let the users pass a `?transaction=3Dtrue` >>>>>> param which indicates they are aware the stream may end early and so >>>>>> would have to paginate from the last emitted key with a skip=3D1. Thi= s >>>>>> will keep the request bodies the same as current CouchDB. However, if= >>>>>> the users got all the data one request, they will end up wasting >>>>>> another request to see if there is more data available. If they didn'= t >>>>>> get any data they might have a too large of a skip value (see [2]) so= >>>>>> would have to guess different values for start/end keys. Or impose ma= x >>>>>> limit for the `skip` parameter. >>>>>>=20 >>>>>> C) End the stream and add a final metadata row like a "transaction": >>>>>> "timeout" at the end. That will let the user know to keep paginating >>>>>> from the last key onward. This won't work for `_all_dbs` and >>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes >>>>>> feeds and only use this for views and and _all_docs? If we like this >>>>>> choice, let's think what happens for those as I couldn't come up with= >>>>>> anything decent there. >>>>>>=20 >>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark >>>>>> "key" of where the iteration stopped and the current "skip" and >>>>>> "limit" params, which would keep decreasing. Then user would pass >>>>>> those in "start_key=3D..." in the next request along with the limit a= nd >>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,= >>>>>> "key":"..."}. This has the same issue with array results for >>>>>> `_all_dbs` and `_dbs_info`[3]. >>>>>>=20 >>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values >>>>>> there such that response time is likely to fit in one transaction. >>>>>> This could be tricky as different runtime environments will have >>>>>> different characteristics. Also, if the timeout happens there isn't a= >>>>>> a nice way to send an HTTP error since we already sent the 200 >>>>>> response. The downside is that this might break how some users use th= e >>>>>> API, if say the are using large skips and limits already. Perhaps her= e >>>>>> we do both B and D, such that if users want transactional behavior, >>>>>> they specify that `transaction=3Dtrue` param and only then we enforce= >>>>>> low limit and skip maximums. >>>>>>=20 >>>>>> F) At least for `_all_docs` it seems providing a point-in-time >>>>>> snapshot view doesn't necessarily need to be tied to transaction >>>>>> boundaries. We could check the update sequence of the database at the= >>>>>> start of the next transaction and if it hasn't changed we can continu= e >>>>>> emitting a consistent view. This can apply to C and D and would just >>>>>> determine when the stream ends. If there are no writes happening to >>>>>> the db, this could potential streams all the data just like option A >>>>>> would do. Not entirely sure if this would work for views. >>>>>>=20 >>>>>> So what do we think? I can see different combinations of options here= , >>>>>> maybe even different for each API point. For example `_all_dbs`, >>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but >>>>>> have parameters to do F, etc. >>>>>>=20 >>>>>> Cheers, >>>>>> -Nick >>>>>>=20 >>>>>> Some footnotes: >>>>>>=20 >>>>>> [1] _changes feeds is the only one that works currently. It behaves a= s >>>>>> per RFC >>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb= -seq-index.md#access-patterns >>> . >>>>>> That is, we continue streaming the data by resetting the transaction >>>>>> object and restarting from the last emitted key (db sequence in this >>>>>> case). However, because the transaction restarts if a document is >>>>>> updated while the streaming take place, it may appear in the _changes= >>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd >>>>>> have to document it, since previously we presented this point-in-time= >>>>>> snapshot of the database from when we started streaming. >>>>>>=20 >>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't >>>>>> currently support efficient offsets for key selectors >>>>>> ( >>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key= -selectors-for-paging >>> ) >>>>>> we implemented skip by iterating over the data. This means that a ski= p >>>>>> of say 100000 could keep timing out the transaction without yielding >>>>>> any data. >>>>>>=20 >>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an >>>>>> obvious place to insert a last metadata row. >>>>>>=20 >>>>>> [4] For example they have a constraint that documents "a" and "z" >>>>>> cannot both be in the database at the same time. But when iterating >>>>>> it's possible that "a" was there at the start. Then by the end, "a" >>>>>> was removed and "z" added, so both "a" and "z" would appear in the >>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the sam= e >>>>>> "relaxed" constrains: >>>>>>=20 >>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality= >>>>>>=20 >>>>=20 >>>>=20 >>>=20 >>=20