From dev-return-49286-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Tue Apr 28 17:15:18 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B97CA180637 for ; Tue, 28 Apr 2020 19:15:17 +0200 (CEST) Received: (qmail 31904 invoked by uid 500); 28 Apr 2020 17:15:16 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 31892 invoked by uid 99); 28 Apr 2020 17:15:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2020 17:15:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E646D18000F for ; Tue, 28 Apr 2020 17:15:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.3 X-Spam-Level: X-Spam-Status: No, score=0.3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id iIxH7QEbhd3A for ; Tue, 28 Apr 2020 17:15:13 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.67; helo=mail-ua1-f67.google.com; envelope-from=vatamane@gmail.com; receiver= Received: from mail-ua1-f67.google.com (mail-ua1-f67.google.com [209.85.222.67]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id A9E4EBB8C5 for ; Tue, 28 Apr 2020 17:15:13 +0000 (UTC) Received: by mail-ua1-f67.google.com with SMTP id c24so22142031uap.13 for ; Tue, 28 Apr 2020 10:15:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=MQ3ezhC9SF+cVNgwDbLXA26gF83Av9a7Sz+ILwUPqDM=; b=g7ztVqMYP57w1j1CnjpUWzi6wF9IA0SFmKydBKglJgbeRzfJ1iWyRbc1xvPIUpz6pT bePCeC4IpB8svxGNnAzVunTsSABON+Q+gIQHDEuKg+712WvkRw8V9h3zSi4BpJCiRLkS WrF2nY+C85tQhgEZEx/gb/kbHLVEV8wUuzkzVPoxtT4fg7LrA/+oMHKNcPb4HivE3ci6 6QxX8sJSz+5KZ1j3i0EXbaUa93AK0o8QdYK8bkyYrlinsf+LkIi534gCV8imUktael1M cDtIp/dq0ui+kElrVUVdjRgppdYJqnYxtQXbJJIPWJVIWP+QOcLxC9hdw/dJHvxqSkeo ezzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=MQ3ezhC9SF+cVNgwDbLXA26gF83Av9a7Sz+ILwUPqDM=; b=IXXbdTg/qNyG9ul3rvAt7iBV9ntMsHfz7OvVNFg7eMpnXOu/5C474mJLnr3c27dmTV n4juxttuJoRGHs5B05PXqFDZ3waFgnCgIpFMJxdpulK3XgreZ0HzrTKtQCy2mNr3wXv0 oWmoH4+VSZi73orJJYLyImfay0Y5xGq8Xvy2G4IcwesoWdT9pyaBhusUKXYq8zlCzmIr scM4n0XXm6qHPvktZr0kpFtXGyiHsU/zJEzlgsCG2iagExvhfpRgOge93jOLuWM5iovZ xNV5SLU0KKR9PnzPAqdbNFUsIoykp60/8/tR/NsewesZGMhaPGSgHTMEClKW7s8I3ykj h3vw== X-Gm-Message-State: AGi0PuaBtuXBLQrKUXxFMuVrraHf9+1aRHcUPQp/rh1VkTEjJaBW/0GX xcaLiYO2X9tPK9pjXj+P736WctF3VGquv3k3zE9Xdw== X-Google-Smtp-Source: APiQypKu/7TjKF+z2tBBCEAop7RmYnlEcSGmejCUItsUFWwSqur9mCHBjE5gXdHndt+Iv7x/DF+hNjhyKPAVpr0iE/A= X-Received: by 2002:ab0:7298:: with SMTP id w24mr22301582uao.95.1588094112621; Tue, 28 Apr 2020 10:15:12 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Nick Vatamaniuc Date: Tue, 28 Apr 2020 13:15:01 -0400 Message-ID: Subject: Re: [DISCUSS] Streaming API in CouchDB 4.0 To: dev@couchdb.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The `{"href": "https://myserver.com/myddb/_all_docs?limit=3D50&descending= =3Dtrue"}` might be tricky if requests have to go through a few reverse proxies before reaching CouchDB. CouchDB might not know its own "external" domain so to speak. I have used X-Forwarded-For before for this exact pattern before, and there is even an RFC for it now https://tools.ietf.org/html/rfc7239, but it can be rather fragile. On `bookmark` vs `token`, I like bookmark, cursor, iterator/iterate better than token I think. Token us used in authentication / authorization so it might get confusing. On the `previous`/`next`/`first` wonder if `previous` and makes sense if we already know the direction they are iterating toward based on the initial request. Or do you propose that the bookmark doesn't encode the direction anymore? Thinking more about this and what's included in the bookmark, it doesn't even need to include the limit. A user could change their limit from request to request even when using a bookmark. If we don't include the direction of iteration, we are down to just the start_key that actually keeps changing in the bookmark and it would be kinda silly to b64 encode the start key by itself in an opaque blob? So going with that, we could just have a matching parameter like inclusive_start=3Dfalse (to match inclusive_end=3D) and let users pick off the key from the result and use that as the start_key with inclusive_start=3Dfalse so they continue iterating from were they stopped. Also I think _dbs_info emits an array as well. We just recently added a GET method for it to match _list_dbs. On Tue, Apr 28, 2020 at 12:38 PM Paul Davis w= rote: > > Seems reasonable to me. I'd agree that setting query string parameters > with a bookmark should be rejected. I was also going to suggest > eliding the href member. In the examples I've seen those are usually > structured as something like: > > "links": { > "previous": "/path/and/qs=3Dfoo", > "next": "/path/and/qs=3Dbar" > } > > On Tue, Apr 28, 2020 at 6:56 AM Ilya Khlopotov wrote: > > > > Hello, > > > > I would like to introduce second proposal. > > > > 1) Add new optional query field called `bookmark` (or `token`) to follo= wing endpoints > > - {db}/_all_docs > > - {db}/_all_docs/queries > > - _dbs_info > > - {db}/_design/{ddoc}/_view/{view} > > - {db}/_design/{ddoc}/_view/{view}/queries > > 2) Add following additional fields into response: > > ``` > > "first": { > > "href": "https://myserver.com/myddb/_all_docs?limit=3D50&descen= ding=3Dtrue" > > }, > > "previous": { > > "href": "https://myserver.com/myddb/_all_docs?bookmark=3D983ui= wfjkdsdf" > > }, > > "next": { > > "href": "https://myserver.com/myddb/_all_docs?bookmark=3D12343t= yekf3" > > }, > > ``` > > 3) Implement per-endpoint configurable max limits > > ``` > > [request_limits] > > _all_docs =3D 5000 > > _all_docs/queries =3D 5000 > > _all_dbs =3D 5000 > > _dbs_info =3D 5000 > > _view =3D 2500 > > _view/queries =3D 2500 > > _find =3D 2500 > > ``` > > 4) Implement following semantics: > > - The bookmark would be opaque token and would include information n= eeded to ensure proper pagination without the need to repeat initial parame= ters of the request. In fact we might prohibit setting additional parameter= s when bookmark query field is specified. > > - don't use delayed responses when `bookmark` field is provided > > - don't use delayed responses when `limit` query key is specified an= d when it is below the max limit > > - return 400 when limit query key is specified and it is greater tha= n the max limit > > - return 400 when we stream rows (in case when `limit` query key was= n't specified) and reach max limit > > - the `previous`/`next`/`first` keys are optional and we omit them f= or the cases they don't make sense > > > > Latter on we would introduce API versioning and deal with `{db}/_change= s` and `_all_docs` endpoints. > > > > Questions: > > - `bookmark` vs `token`? > > - should we prohibit setting other fields when bookmark is set? > > - `previous`/`next`/`first` as href vs token value itself (i.e. `{"prev= ious": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`) > > > > Best regards, > > iilyak > > > > On 2020/04/22 20:18:57, Ilya Khlopotov wrote: > > > Hello everyone, > > > > > > Based on the discussions on the thread I would like to propose a numb= er of first steps: > > > 1) introduce new endpoints > > > - {db}/_all_docs/page > > > - {db}/_all_docs/queries/page > > > - _all_dbs/page > > > - _dbs_info/page > > > - {db}/_design/{ddoc}/_view/{view}/page > > > - {db}/_design/{ddoc}/_view/{view}/queries/page > > > - {db}/_find/page > > > > > > These new endpoints would act as follows: > > > - don't use delayed responses > > > - return object with following structure > > > ``` > > > { > > > "total": Total, > > > "bookmark": base64 encoded opaque value, > > > "completed": true | false, > > > "update_seq": when available, > > > "page": current page number, > > > "items": [ > > > ] > > > } > > > ``` > > > - the bookmark would include following data (base64 or protobuff???): > > > - direction > > > - page > > > - descending > > > - endkey > > > - endkey_docid > > > - inclusive_end > > > - startkey > > > - startkey_docid > > > - last_key > > > - update_seq > > > - timestamp > > > ``` > > > > > > 2) Implement per-endpoint configurable max limits > > > ``` > > > _all_docs =3D 5000 > > > _all_docs/queries =3D 5000 > > > _all_dbs =3D 5000 > > > _dbs_info =3D 5000 > > > _view =3D 2500 > > > _view/queries =3D 2500 > > > _find =3D 2500 > > > ``` > > > > > > Latter (after few years) CouchDB would deprecate and remove old endpo= ints. > > > > > > Best regards, > > > iilyak > > > > > > On 2020/02/19 22:39:45, Nick Vatamaniuc wrote: > > > > Hello everyone, > > > > > > > > I'd like to discuss the shape and behavior of streaming APIs for Co= uchDB 4.x > > > > > > > > By "streaming APIs" I mean APIs which stream data in row as it gets > > > > read from the database. These are the endpoints I was thinking of: > > > > > > > > _all_docs, _all_dbs, _dbs_info and query results > > > > > > > > I want to focus on what happens when FoundationDB transactions > > > > time-out after 5 seconds. Currently, all those APIs except _changes= [1] > > > > feeds, will crash or freeze. The reason is because the > > > > transaction_too_old error at the end of 5 seconds is retry-able by > > > > default, so the request handlers run again and end up shoving the > > > > whole request down the socket again, headers and all, which is > > > > obviously broken and not what we want. > > > > > > > > There are few alternatives discussed in couchdb-dev channel. I'll > > > > present some behaviors but feel free to add more. Some ideas might > > > > have been discounted on the IRC discussion already but I'll present > > > > them anyway in case is sparks further conversation: > > > > > > > > A) Do what _changes[1] feeds do. Start a new transaction and contin= ue > > > > streaming the data from the next key after last emitted in the > > > > previous transaction. Document the API behavior change that it may > > > > present a view of the data is never a point-in-time[4] snapshot of = the > > > > DB. > > > > > > > > - Keeps the API shape the same as CouchDB <4.0. Client libraries > > > > don't have to change to continue using these CouchDB 4.0 endpoints > > > > - This is the easiest to implement since it would re-use the > > > > implementation for _changes feed (an extra option passed to the fol= d > > > > function). > > > > - Breaks API behavior if users relied on having a point-in-time[4] > > > > snapshot view of the data. > > > > > > > > B) Simply end the stream. Let the users pass a `?transaction=3Dtrue= ` > > > > param which indicates they are aware the stream may end early and s= o > > > > would have to paginate from the last emitted key with a skip=3D1. T= his > > > > will keep the request bodies the same as current CouchDB. However, = if > > > > the users got all the data one request, they will end up wasting > > > > another request to see if there is more data available. If they did= n't > > > > get any data they might have a too large of a skip value (see [2]) = so > > > > would have to guess different values for start/end keys. Or impose = max > > > > limit for the `skip` parameter. > > > > > > > > C) End the stream and add a final metadata row like a "transaction"= : > > > > "timeout" at the end. That will let the user know to keep paginatin= g > > > > from the last key onward. This won't work for `_all_dbs` and > > > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes > > > > feeds and only use this for views and and _all_docs? If we like thi= s > > > > choice, let's think what happens for those as I couldn't come up wi= th > > > > anything decent there. > > > > > > > > D) Same as C but to solve the issue with skips[2], emit a bookmark > > > > "key" of where the iteration stopped and the current "skip" and > > > > "limit" params, which would keep decreasing. Then user would pass > > > > those in "start_key=3D..." in the next request along with the limit= and > > > > skip params. So something like "continuation":{"skip":599, "limit":= 5, > > > > "key":"..."}. This has the same issue with array results for > > > > `_all_dbs` and `_dbs_info`[3]. > > > > > > > > E) Enforce low `limit` and `skip` parameters. Enforce maximum value= s > > > > there such that response time is likely to fit in one transaction. > > > > This could be tricky as different runtime environments will have > > > > different characteristics. Also, if the timeout happens there isn't= a > > > > a nice way to send an HTTP error since we already sent the 200 > > > > response. The downside is that this might break how some users use = the > > > > API, if say the are using large skips and limits already. Perhaps h= ere > > > > we do both B and D, such that if users want transactional behavior, > > > > they specify that `transaction=3Dtrue` param and only then we enfor= ce > > > > low limit and skip maximums. > > > > > > > > F) At least for `_all_docs` it seems providing a point-in-time > > > > snapshot view doesn't necessarily need to be tied to transaction > > > > boundaries. We could check the update sequence of the database at t= he > > > > start of the next transaction and if it hasn't changed we can conti= nue > > > > emitting a consistent view. This can apply to C and D and would jus= t > > > > determine when the stream ends. If there are no writes happening to > > > > the db, this could potential streams all the data just like option = A > > > > would do. Not entirely sure if this would work for views. > > > > > > > > So what do we think? I can see different combinations of options he= re, > > > > maybe even different for each API point. For example `_all_dbs`, > > > > `_dbs_info` are always A, and `_all_docs` and views default to A bu= t > > > > have parameters to do F, etc. > > > > > > > > Cheers, > > > > -Nick > > > > > > > > Some footnotes: > > > > > > > > [1] _changes feeds is the only one that works currently. It behaves= as > > > > per RFC https://github.com/apache/couchdb-documentation/blob/master= /rfcs/003-fdb-seq-index.md#access-patterns. > > > > That is, we continue streaming the data by resetting the transactio= n > > > > object and restarting from the last emitted key (db sequence in thi= s > > > > case). However, because the transaction restarts if a document is > > > > updated while the streaming take place, it may appear in the _chang= es > > > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'= d > > > > have to document it, since previously we presented this point-in-ti= me > > > > snapshot of the database from when we started streaming. > > > > > > > > [2] Our streaming APIs have both skips and limits. Since FDB doesn'= t > > > > currently support efficient offsets for key selectors > > > > (https://apple.github.io/foundationdb/known-limitations.html#dont-u= se-key-selectors-for-paging) > > > > we implemented skip by iterating over the data. This means that a s= kip > > > > of say 100000 could keep timing out the transaction without yieldin= g > > > > any data. > > > > > > > > [3] _all_dbs and _dbs_info return a JSON array so they don't have a= n > > > > obvious place to insert a last metadata row. > > > > > > > > [4] For example they have a constraint that documents "a" and "z" > > > > cannot both be in the database at the same time. But when iterating > > > > it's possible that "a" was there at the start. Then by the end, "a" > > > > was removed and "z" added, so both "a" and "z" would appear in the > > > > emitted stream. Note that FoundationDB has APIs which exhibit the s= ame > > > > "relaxed" constrains: > > > > https://apple.github.io/foundationdb/api-python.html#module-fdb.loc= ality > > > > > > >