From dev-return-41266-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Wed Oct 31 15:56:39 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id ED6CB18065D for ; Wed, 31 Oct 2018 15:56:38 +0100 (CET) Received: (qmail 92797 invoked by uid 500); 31 Oct 2018 14:56:37 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 92785 invoked by uid 99); 31 Oct 2018 14:56:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Oct 2018 14:56:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id DB5BEC1EB3 for ; Wed, 31 Oct 2018 14:56:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.889 X-Spam-Level: * X-Spam-Status: No, score=1.889 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id xACQ5aFjyiKb for ; Wed, 31 Oct 2018 14:56:35 +0000 (UTC) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0BE015F5B1 for ; Wed, 31 Oct 2018 14:56:35 +0000 (UTC) Received: by mail-lj1-f169.google.com with SMTP id q186-v6so1764906ljb.5 for ; Wed, 31 Oct 2018 07:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:from:subject:date:importance; bh=1tMCQcV2mgjpIfAI/BrxUOGfLBRL3Mz67fm0yywbrXY=; b=VyKuEFbce28fL9zpWTaedGU55etFa0YQtOfPXvVD5gNM0JXbmcTjK1j8MVsPtjqhnA c3IUKWhfobuj8wvw/gqJnOeaWlWuGXXhrZ4u2V6PM4TaVqKwO1eyqEWuUXe6NG/haVQI tECyPAVYTHUcHdFTWk2DeoVS4TkpHdTth1Rqs0Htre8i/TPSKW1b2wp0CwcUYMhLQGTo 5CRXENGHyL4iCYhFDQRA/egB0Pf8gKJ8gEv+2xZPUdCYwipAdZ5OmalyQXRIPyxXLyj6 gxvLVrEB0U4VYW26YXUyOqL7tFpGH77pKQO0vT02AhFQPRkQeyh7t9iKK8YXdJH7K9j1 4GtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:from:subject:date :importance; bh=1tMCQcV2mgjpIfAI/BrxUOGfLBRL3Mz67fm0yywbrXY=; b=c5hElkdN+73+azpeVg4MfhxONYi6t/MXkMP2hNkNILzQL5Ie7UWN68KjJ06KJxNuzT 9sD5BrIPmlr4m+T02dycCowa3Xf6703e6CcpXv+Ja/puIS7x1BYjshxicf1eHDvdt5i3 71su+SgnwwSPT5JtuR8OaU38eKFaVLe81gjRvITB32mMGaJMMCtCRHMPvTVUal2gQhoT n6WzhikXhfzSikWKgnCXnuM9yjYzSVSvI3Lf8pF9W12+GT57pCp02d3agi2gZeCaUOTe HcoMK3htVns2KU/ekk3fx0mk8xIQqeTYx8y3tDh37+JEVURo72ONtfCrU58Cxa1eOEUx qYNA== X-Gm-Message-State: AGRZ1gLCmtZLVhrTfjbUqVDhnH65Duf2aHwAS2s32vo9/v84kZIi63ZW ZEP/gFbzeQclnbGB7s562GStVreL X-Google-Smtp-Source: AJdET5fKZV6KWdPUi1fvJjwr3zRp32SQ7nkOwYZtHmNKpShx3w6F0mOIJIwbEv6gl6pkdNf6SEYJqg== X-Received: by 2002:a2e:5b93:: with SMTP id m19-v6mr1964939lje.115.1540997793955; Wed, 31 Oct 2018 07:56:33 -0700 (PDT) Received: from ?IPv6:::ffff:172.25.4.36? ([195.239.208.174]) by smtp.gmail.com with ESMTPSA id k18-v6sm4076430ljk.58.2018.10.31.07.56.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Oct 2018 07:56:33 -0700 (PDT) Message-ID: <5bd9c2a1.1c69fb81.c43a9.da77@mx.google.com> MIME-Version: 1.0 To: "dev@ignite.apache.org" From: Stanislav Lukyanov Subject: Partition Loss Policies issues Date: Wed, 31 Oct 2018 17:56:32 +0300 Importance: normal X-Priority: 3 Content-Type: multipart/alternative; boundary="_491E3854-9ECF-4042-94FB-E95EBB2F8BA2_" --_491E3854-9ECF-4042-94FB-E95EBB2F8BA2_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi Igniters, I've been looking into various scenarios of Partition Loss Policies usage r= ecently, and found a number of issues in the current implementation. I'll start with an overview, but if you'd like to dive to a proposal I have= right now then please feel free to scroll down to TLDR. The list of issues is below: https://issues.apache.org/jira/browse/IGNITE-10041: Partition loss policies= work incorrectly with BLT https://issues.apache.org/jira/browse/IGNITE-10043: Lost partitions list is= reset when only one server is alive in the cluster https://issues.apache.org/jira/browse/IGNITE-9841: SQL doesn't take lost pa= rtitions into account when persistence is enabled https://issues.apache.org/jira/browse/IGNITE-10057: SQL queries hang during= rebalance if there are LOST partitions https://issues.apache.org/jira/browse/IGNITE-9902: ScanQuery doesn't take l= ost partitions into account https://issues.apache.org/jira/browse/IGNITE-10059: Local scan query agains= t LOST partition fails https://issues.apache.org/jira/browse/IGNITE-10044: LOST partition is marke= d as OWNING after the owner rejoins with existing persistent data https://issues.apache.org/jira/browse/IGNITE-10058: resetLostPartitions() l= eaves an additional copy of a partition in the cluster I'm sure this is not a complete list, but this is what I could find by tack= ling how queries and=20 persistence interact with current handling of partition loss. It seems that the issues - from this list and some other fixed recently - c= an be split into three categories - corner case bugs - there are always some, and we can fix them as they sho= w up - handling of lost partitions by different APIs - while JCache API handles = lost partitions fine,=20 SQL and Scan queries have known issues; other APIs, such different types of= queries, DataStreamer, etc probably need to have more testing - Partition Loss Policices + BLT (https://issues.apache.org/jira/browse/IGN= ITE-10041) - BLT seems=20 to be fundametally conflicting with the pre-existing semantics of partition= loss While the former two categories can be solved case-by-case, the last one ne= eds a wider design effort. We need to reimagine our partition loss semantics for BLT, and change behav= ior accordingly. For now, most of the features don't really work for a cache with BLT, with = only READ_WRITE_SAFE and READ_ONLY_SAFE working correctly (good thing - these two are the most usefu= l policies anyway). TLDR: we have issues with partition loss policices, and the largest one is = that BLT semantics conflict with most partition lost policices, and we need to address this so= mehow. What I suggest to do right now: 1. Deny the configurations that don't work - e.g. just throw an exception i= f a cache starts with BLT and PartitionLossPolicy.IGNORE or others. 2. Change default PartitionLossPolicy to READ_WRITE_SAFE *for persistent ca= ches only*. This is what effectively in place for the persistent caches already (since = IGNORE semantics are not supported), so this shouldn't bring a lot of compatibility issues. I believe doing this will at least help us to protect the users from unexpe= cted/inconsistent behavior. Actual design changes can be done later, e.g. as a part of IEP 4 Phase 2/3. WDYT? Thanks, Stan --_491E3854-9ECF-4042-94FB-E95EBB2F8BA2_--