From dev-return-49438-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Mon Jul 13 20:15:29 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EE82618037A for ; Mon, 13 Jul 2020 22:15:28 +0200 (CEST) Received: (qmail 75298 invoked by uid 500); 13 Jul 2020 20:15:28 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 75285 invoked by uid 99); 13 Jul 2020 20:15:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2020 20:15:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2E11118140C for ; Mon, 13 Jul 2020 20:15:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.201 X-Spam-Level: X-Spam-Status: No, score=-0.201 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Os-Amauv8jSc for ; Mon, 13 Jul 2020 20:15:25 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.46; helo=mail-ua1-f46.google.com; envelope-from=vatamane@gmail.com; receiver= Received: from mail-ua1-f46.google.com (mail-ua1-f46.google.com [209.85.222.46]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 6E93CBE314 for ; Mon, 13 Jul 2020 20:15:25 +0000 (UTC) Received: by mail-ua1-f46.google.com with SMTP id n4so4584790uae.5 for ; Mon, 13 Jul 2020 13:15:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=O5k/fmHu0KxNatmG8IzpV0RlV1++ACkzC1kboPf8XHo=; b=ri6VoB9Ty9rdrf4K20H9GWBSQaGLCcHTDy67SZ0i3CrGq87OjGOaQiRx/w7wPG+fD4 qToFIe5tPht2AUmNnqBm8kbNLtZcZlTDbsG46F2TTiwW2udpank+/l/PRMH8Il93NXSn gN7v2pZ4F1bR9Hs3NEr1IX5ZNYQdrbfcBNBHiUnrte9jbfGL2BqEw1cfQwmsdxLGamwT FsX2HlbUZX1gx7P69NkVvlLfO6tO4bNuHE7xTaKyZm9p4DHw5fSAHyduXkeSpCjoGkEy Pa7EbLo7mo6xgK8xBxvKdbgf4y6xQG06e9z/bXIdk+xxsBFbSG7GC21D8Us1cNZgtTO3 q1Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=O5k/fmHu0KxNatmG8IzpV0RlV1++ACkzC1kboPf8XHo=; b=Ed0yfDtfGSH8qg+kVagn8YEURIAh2Rh9nusALDoTQ1rfZ155qu5XRCb7q2ANzoL4oi U8WDqJlyZ0UVAHwvxkpwsSHLvqZFp2kvtk9p1JFOl0WlOTN8TrWUikCmQBtF+qO6LDKM kCZjTJuNk+wY9Qxxi4hcF0BIutUwLenIt4wEdk3cd/xw/yKfBXnUxlQmuPaM2YxQ+3XE +CKlqJ6pLZJr+wWvzG1TY3ucwcTd3RfBWbFbG2+fyi97hPx4VVdMOaBCrp+9rsQ4cL76 4lA3Ul415fKfbMH8d0sOZAePGBcvsMVTO6yxGNEhsq3lwVAizmXkIIRVK7pnrIb7ndZb gEvg== X-Gm-Message-State: AOAM533gNh8a/MiQC7rz0J/2tK5/NzrJT8E1DBfj4QGXauhn8PX2A/JB c4nZtmZARTqNWupsZ0ulwiwYwJce5Z8hHP3r1vKIVx6E X-Google-Smtp-Source: ABdhPJyAz2AoLhNrc9UWUxRcPaFVPn9C3XdAuGEWqbhX/+Q/NQ+gi0ODbk3XUptaaxWqV/bCz74ssnYpbd6DXW6XARQ= X-Received: by 2002:ab0:29cb:: with SMTP id i11mr1086423uaq.12.1594671318723; Mon, 13 Jul 2020 13:15:18 -0700 (PDT) MIME-Version: 1.0 References: <1DEAF31A-0B68-4688-981C-F328F802AFD9@apache.org> In-Reply-To: <1DEAF31A-0B68-4688-981C-F328F802AFD9@apache.org> From: Nick Vatamaniuc Date: Mon, 13 Jul 2020 16:15:07 -0400 Message-ID: Subject: Re: [DISCUSS] couchdb 4.0 transactional semantics To: dev@couchdb.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for bringing the topic up for the discussion! For background, this topic was discussed on the mailing list starting in February, 2019 https://lists.apache.org/thread.html/r02cee7045cac4722e1682bb69ba0ec791f5cc= e025597d0099fb34033%40%3Cdev.couchdb.apache.org%3E The primary reason for restart_tx option is to provide compatibility for _changes feeds to allow older replicators to handle 4.0 sources. It starts a new transaction after 5 seconds or so (a current FDB limitation, might go up in the future) and transparently continues to stream data where it left off. Ex, streaming [a,b,c,d], times out after b, then it will continue with c, d etc. Currently this is also used for other streaming APIs as an alternative to returning mangled JSON after emitting a 200 response and streaming some of the rows. However it is not used for paginated responses, the new APIs developed by Ilya. So users have an option to get the guaranteed snapshot behavior option as well. And for completeness, if we decide to remove the option, we should specify what happens if we remove it and get a transaction_too_old exception. Currently the behavior would be to restart the transaction, resend all the headers and all the rows again down the socket, which I don't think anyone wants, but is what we'd get if we just make {restart_tx, false} > I understand that automatically resetting the FDB txn during a response = is an attempt to work around that and maintain "compatibility" with CouchDB= < 4 semantics. I think it fails to do so and is very misleading. It is a trade-off in order to keep the same API shape as before. Sure, streaming all the docs with _all_docs or _changes feeds is not a great pattern but many applications are implemented that way already. Letting them migrate to 4.0 without having to rewrite the application with the caveat that they might see a document updated in the _all_docs stream after the request has already started, is a nicer choice, I think, than forcing them to rewrite their application, which could lead to a python 2/3 scenario. Due to having multiple shards (Q>1), as discussed in the original mailing thread by Mike (https://lists.apache.org/thread.html/r8345f534a6fa88c107c1085fba13e660e0e2= aedfd206c2748e002664%40%3Cdev.couchdb.apache.org%3E), we don't provide a strict read-only snapshot guarantee in 2.x and 3.x anyway, so users would have to handle scenarios where a document might appear in the stream that wasn't there at the start of the request already. Though, granted, a much smaller corner case but I wonder how many users care to handle that... Currently users do have an option of using the new paginated API which disables restart_tx behavior https://github.com/apache/couchdb/blob/prototype/fdb-layer/src/chttpd/src/c= httpd_db.erl#L947, though I am not sure what happens when transaction_too_old exception is thrown then (emit a bookmark?) So based on the compatibility consideration, I'd vote to keep the restart_tx option (configurable perhaps if we figure out what to do when it is disabled) in order to allow users to migrate their application to 4.0. At least informally we promised users to keep a strong API compatibility when we released 3.0 with an eye towards 4.0 (https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0/). I'd think not emitting all the data in a _changes or _all_docs response would break that compatibility more than using multiple transactions. As for what happens when a transaction_too_old is thrown, I could see an option passed in, something like, single_snapshot=3Dtrue, and then use Adam's suggestion to accumulate all the rows in memory and if we hit the end of the transaction return a 400 error. We won't emit anything out while rows are accumulated, so users don't get partial data, it will be every row requested or a 400 error (so no chance of perceived data loss). Users may retry if they think it was a temporary hiccup or may use a small limit number. Cheers, -Nick On Mon, Jul 13, 2020 at 2:05 PM Robert Samuel Newson w= rote: > > Hi All, > > I'm concerned to see the restart_fold function in fabric2_fdb (https://gi= thub.com/apache/couchdb/blob/prototype/fdb-layer/src/fabric/src/fabric2_fdb= .erl#L1828) in the 4.0 development branch. > > The upshot of doing this is that a CouchDB response could be taken across= multiple snapshots of the database, which is not the behaviour of CouchDB = 1 through 3. > > I don't think this is ok (with the obvious and established exception of a= continuous changes feed, where new snapshots are continuously visible at t= he end of the response). > > FoundationDB imposes certain limits on transactions, the most notable bei= ng the 5 second maximum duration. I understand that automatically resetting= the FDB txn during a response is an attempt to work around that and mainta= in "compatibility" with CouchDB < 4 semantics. I think it fails to do so an= d is very misleading. > > Discuss. > > B. >