From dev-return-49294-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Thu Apr 30 15:16:44 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 099DF180630 for ; Thu, 30 Apr 2020 17:16:43 +0200 (CEST) Received: (qmail 95007 invoked by uid 500); 30 Apr 2020 15:16:43 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 94995 invoked by uid 99); 30 Apr 2020 15:16:43 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Apr 2020 15:16:43 +0000 Received: from auth2-smtp.messagingengine.com (auth2-smtp.messagingengine.com [66.111.4.228]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 099F71005 for ; Thu, 30 Apr 2020 15:16:43 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailauth.nyi.internal (Postfix) with ESMTP id D56FB27C0054 for ; Thu, 30 Apr 2020 11:16:42 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 30 Apr 2020 11:16:42 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrieehgdekhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecuogfuuhhsphgvtghtffhomhgrihhnucdlgeelmdenuc fjughrpefhtgfgggfuffhfvfgjkffosehtqhhmtdhhtdejnecuhfhrohhmpeetuggrmhcu mfhotgholhhoshhkihcuoehkohgtohhlohhskhesrghprggthhgvrdhorhhgqeenucggtf frrghtthgvrhhnpeevgeekuedvveejgeeugffgteeigefftddvkeeiieefiefhhfefheek ueeffeekheenucffohhmrghinhepghhithhhuhgsrdgtohhmpdhmhihsvghrvhgvrhdrtg homhdpghhithhhuhgsrdhiohenucfkphepvdegrdeitddrudelfedrudekieenucevlhhu shhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehkohgtohhlohhskh domhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqleekvdeigeehiedtqdduheeh ieejjeehgedqkhhotgholhhoshhkpeeprghprggthhgvrdhorhhgsehfrghsthhmrghilh drtghomh X-ME-Proxy: Received: from [10.0.0.178] (c-24-60-193-186.hsd1.ma.comcast.net [24.60.193.186]) by mail.messagingengine.com (Postfix) with ESMTPA id 7249A3065F33 for ; Thu, 30 Apr 2020 11:16:42 -0400 (EDT) From: Adam Kocoloski Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: [DISCUSS] Streaming API in CouchDB 4.0 Date: Thu, 30 Apr 2020 11:16:41 -0400 References: To: dev@couchdb.apache.org In-Reply-To: Message-Id: <518742C0-820A-4568-9645-DECBBF9069C8@apache.org> X-Mailer: Apple Mail (2.3608.80.23.2.2) I think this is a good reason to fall back to just including the value = of the bookmark in =E2=80=9Cfirst=E2=80=9D, =E2=80=9Cnext=E2=80=9D and = =E2=80=9Clast=E2=80=9D, and then leaving it up to the client to decide = whether to supply the bookmark in the URL or in the request body. Adam > On Apr 30, 2020, at 10:47 AM, Joan Touzet wrote: >=20 > Can we keep a distributed hash of doc keys server side with a smaller = handle we hand to clients? >=20 > If the cache can't be found or a restart happens, oh well. >=20 > -Joan >=20 > On 2020-04-30 10:23, Ilya Khlopotov wrote: >> There is a problem with representing `next`/`previous`/`first` as = path. With 5kB sized doc keys and we could exceed max URL length (8192 = bytes). This means we would have to support POST. The question is how to = handle the case when the URL is greater than 8192. The problem is = CouchDB don't knows it's DNS name so we don't know the safe value to = compare with. >> Options are: >> 1) always use POST for pagination >> 2) add server_name to config and return error when bookmark length = exceeds dynamically calculated threshold. The threshold would account = for db name length, server_name length, port and scheme length. >> - what error to return? >> I think option number 2 is too subtle to implement. >> The downside of option 1 is it is a bit harder to use from the = browser or curl. >> On 2020/04/29 17:27:59, Ilya Khlopotov wrote: >>> I think I addressed all comments and created an RFC = https://github.com/apache/couchdb-documentation/pull/530 >>>=20 >>> On 2020/04/28 11:56:15, Ilya Khlopotov wrote: >>>> Hello, >>>>=20 >>>> I would like to introduce second proposal. >>>>=20 >>>> 1) Add new optional query field called `bookmark` (or `token`) to = following endpoints >>>> - {db}/_all_docs >>>> - {db}/_all_docs/queries >>>> - _dbs_info >>>> - {db}/_design/{ddoc}/_view/{view} >>>> - {db}/_design/{ddoc}/_view/{view}/queries >>>> 2) Add following additional fields into response: >>>> ``` >>>> "first": { >>>> "href": = "https://myserver.com/myddb/_all_docs?limit=3D50&descending=3Dtrue" >>>> }, >>>> "previous": { >>>> "href": = "https://myserver.com/myddb/_all_docs?bookmark=3D983uiwfjkdsdf" >>>> }, >>>> "next": { >>>> "href": = "https://myserver.com/myddb/_all_docs?bookmark=3D12343tyekf3" >>>> }, >>>> ``` >>>> 3) Implement per-endpoint configurable max limits >>>> ``` >>>> [request_limits] >>>> _all_docs =3D 5000 >>>> _all_docs/queries =3D 5000 >>>> _all_dbs =3D 5000 >>>> _dbs_info =3D 5000 >>>> _view =3D 2500 >>>> _view/queries =3D 2500 >>>> _find =3D 2500 >>>> ``` >>>> 4) Implement following semantics: >>>> - The bookmark would be opaque token and would include = information needed to ensure proper pagination without the need to = repeat initial parameters of the request. In fact we might prohibit = setting additional parameters when bookmark query field is specified. >>>> - don't use delayed responses when `bookmark` field is provided >>>> - don't use delayed responses when `limit` query key is = specified and when it is below the max limit >>>> - return 400 when limit query key is specified and it is greater = than the max limit >>>> - return 400 when we stream rows (in case when `limit` query key = wasn't specified) and reach max limit >>>> - the `previous`/`next`/`first` keys are optional and we omit = them for the cases they don't make sense >>>>=20 >>>> Latter on we would introduce API versioning and deal with = `{db}/_changes` and `_all_docs` endpoints. >>>> Questions: >>>> - `bookmark` vs `token`? >>>> - should we prohibit setting other fields when bookmark is set? >>>> - `previous`/`next`/`first` as href vs token value itself (i.e. = `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": = "iekjhfwo034"}`) >>>>=20 >>>> Best regards, >>>> iilyak >>>>=20 >>>> On 2020/04/22 20:18:57, Ilya Khlopotov wrote: >>>>> Hello everyone, >>>>>=20 >>>>> Based on the discussions on the thread I would like to propose a = number of first steps: >>>>> 1) introduce new endpoints >>>>> - {db}/_all_docs/page >>>>> - {db}/_all_docs/queries/page >>>>> - _all_dbs/page >>>>> - _dbs_info/page >>>>> - {db}/_design/{ddoc}/_view/{view}/page >>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page >>>>> - {db}/_find/page >>>>>=20 >>>>> These new endpoints would act as follows: >>>>> - don't use delayed responses >>>>> - return object with following structure >>>>> ``` >>>>> { >>>>> "total": Total, >>>>> "bookmark": base64 encoded opaque value, >>>>> "completed": true | false, >>>>> "update_seq": when available, >>>>> "page": current page number, >>>>> "items": [ >>>>> ] >>>>> } >>>>> ``` >>>>> - the bookmark would include following data (base64 or = protobuff???): >>>>> - direction >>>>> - page >>>>> - descending >>>>> - endkey >>>>> - endkey_docid >>>>> - inclusive_end >>>>> - startkey >>>>> - startkey_docid >>>>> - last_key >>>>> - update_seq >>>>> - timestamp >>>>> ``` >>>>>=20 >>>>> 2) Implement per-endpoint configurable max limits >>>>> ``` >>>>> _all_docs =3D 5000 >>>>> _all_docs/queries =3D 5000 >>>>> _all_dbs =3D 5000 >>>>> _dbs_info =3D 5000 >>>>> _view =3D 2500 >>>>> _view/queries =3D 2500 >>>>> _find =3D 2500 >>>>> ``` >>>>>=20 >>>>> Latter (after few years) CouchDB would deprecate and remove old = endpoints. >>>>>=20 >>>>> Best regards, >>>>> iilyak >>>>>=20 >>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc = wrote: >>>>>> Hello everyone, >>>>>>=20 >>>>>> I'd like to discuss the shape and behavior of streaming APIs for = CouchDB 4.x >>>>>>=20 >>>>>> By "streaming APIs" I mean APIs which stream data in row as it = gets >>>>>> read from the database. These are the endpoints I was thinking = of: >>>>>>=20 >>>>>> _all_docs, _all_dbs, _dbs_info and query results >>>>>>=20 >>>>>> I want to focus on what happens when FoundationDB transactions >>>>>> time-out after 5 seconds. Currently, all those APIs except = _changes[1] >>>>>> feeds, will crash or freeze. The reason is because the >>>>>> transaction_too_old error at the end of 5 seconds is retry-able = by >>>>>> default, so the request handlers run again and end up shoving the >>>>>> whole request down the socket again, headers and all, which is >>>>>> obviously broken and not what we want. >>>>>>=20 >>>>>> There are few alternatives discussed in couchdb-dev channel. I'll >>>>>> present some behaviors but feel free to add more. Some ideas = might >>>>>> have been discounted on the IRC discussion already but I'll = present >>>>>> them anyway in case is sparks further conversation: >>>>>>=20 >>>>>> A) Do what _changes[1] feeds do. Start a new transaction and = continue >>>>>> streaming the data from the next key after last emitted in the >>>>>> previous transaction. Document the API behavior change that it = may >>>>>> present a view of the data is never a point-in-time[4] snapshot = of the >>>>>> DB. >>>>>>=20 >>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries >>>>>> don't have to change to continue using these CouchDB 4.0 = endpoints >>>>>> - This is the easiest to implement since it would re-use the >>>>>> implementation for _changes feed (an extra option passed to the = fold >>>>>> function). >>>>>> - Breaks API behavior if users relied on having a = point-in-time[4] >>>>>> snapshot view of the data. >>>>>>=20 >>>>>> B) Simply end the stream. Let the users pass a = `?transaction=3Dtrue` >>>>>> param which indicates they are aware the stream may end early and = so >>>>>> would have to paginate from the last emitted key with a skip=3D1. = This >>>>>> will keep the request bodies the same as current CouchDB. = However, if >>>>>> the users got all the data one request, they will end up wasting >>>>>> another request to see if there is more data available. If they = didn't >>>>>> get any data they might have a too large of a skip value (see = [2]) so >>>>>> would have to guess different values for start/end keys. Or = impose max >>>>>> limit for the `skip` parameter. >>>>>>=20 >>>>>> C) End the stream and add a final metadata row like a = "transaction": >>>>>> "timeout" at the end. That will let the user know to keep = paginating >>>>>> from the last key onward. This won't work for `_all_dbs` and >>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes >>>>>> feeds and only use this for views and and _all_docs? If we like = this >>>>>> choice, let's think what happens for those as I couldn't come up = with >>>>>> anything decent there. >>>>>>=20 >>>>>> D) Same as C but to solve the issue with skips[2], emit a = bookmark >>>>>> "key" of where the iteration stopped and the current "skip" and >>>>>> "limit" params, which would keep decreasing. Then user would pass >>>>>> those in "start_key=3D..." in the next request along with the = limit and >>>>>> skip params. So something like "continuation":{"skip":599, = "limit":5, >>>>>> "key":"..."}. This has the same issue with array results for >>>>>> `_all_dbs` and `_dbs_info`[3]. >>>>>>=20 >>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum = values >>>>>> there such that response time is likely to fit in one = transaction. >>>>>> This could be tricky as different runtime environments will have >>>>>> different characteristics. Also, if the timeout happens there = isn't a >>>>>> a nice way to send an HTTP error since we already sent the 200 >>>>>> response. The downside is that this might break how some users = use the >>>>>> API, if say the are using large skips and limits already. Perhaps = here >>>>>> we do both B and D, such that if users want transactional = behavior, >>>>>> they specify that `transaction=3Dtrue` param and only then we = enforce >>>>>> low limit and skip maximums. >>>>>>=20 >>>>>> F) At least for `_all_docs` it seems providing a point-in-time >>>>>> snapshot view doesn't necessarily need to be tied to transaction >>>>>> boundaries. We could check the update sequence of the database at = the >>>>>> start of the next transaction and if it hasn't changed we can = continue >>>>>> emitting a consistent view. This can apply to C and D and would = just >>>>>> determine when the stream ends. If there are no writes happening = to >>>>>> the db, this could potential streams all the data just like = option A >>>>>> would do. Not entirely sure if this would work for views. >>>>>>=20 >>>>>> So what do we think? I can see different combinations of options = here, >>>>>> maybe even different for each API point. For example `_all_dbs`, >>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A = but >>>>>> have parameters to do F, etc. >>>>>>=20 >>>>>> Cheers, >>>>>> -Nick >>>>>>=20 >>>>>> Some footnotes: >>>>>>=20 >>>>>> [1] _changes feeds is the only one that works currently. It = behaves as >>>>>> per RFC = https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-s= eq-index.md#access-patterns. >>>>>> That is, we continue streaming the data by resetting the = transaction >>>>>> object and restarting from the last emitted key (db sequence in = this >>>>>> case). However, because the transaction restarts if a document is >>>>>> updated while the streaming take place, it may appear in the = _changes >>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and = we'd >>>>>> have to document it, since previously we presented this = point-in-time >>>>>> snapshot of the database from when we started streaming. >>>>>>=20 >>>>>> [2] Our streaming APIs have both skips and limits. Since FDB = doesn't >>>>>> currently support efficient offsets for key selectors >>>>>> = (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-= selectors-for-paging) >>>>>> we implemented skip by iterating over the data. This means that a = skip >>>>>> of say 100000 could keep timing out the transaction without = yielding >>>>>> any data. >>>>>>=20 >>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have = an >>>>>> obvious place to insert a last metadata row. >>>>>>=20 >>>>>> [4] For example they have a constraint that documents "a" and "z" >>>>>> cannot both be in the database at the same time. But when = iterating >>>>>> it's possible that "a" was there at the start. Then by the end, = "a" >>>>>> was removed and "z" added, so both "a" and "z" would appear in = the >>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the = same >>>>>> "relaxed" constrains: >>>>>> = https://apple.github.io/foundationdb/api-python.html#module-fdb.locality >>>>>>=20 >>>>>=20 >>>>=20 >>>=20