From dev-return-44997-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Wed Feb 27 16:53:28 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 367AE180608 for ; Wed, 27 Feb 2019 17:53:28 +0100 (CET) Received: (qmail 79787 invoked by uid 500); 27 Feb 2019 16:53:27 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 79768 invoked by uid 99); 27 Feb 2019 16:53:26 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2019 16:53:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 25D70C8CB9 for ; Wed, 27 Feb 2019 16:53:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.112 X-Spam-Level: * X-Spam-Status: No, score=1.112 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id EAp8p7FGZk4E for ; Wed, 27 Feb 2019 16:53:21 +0000 (UTC) Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 91DEE62400 for ; Wed, 27 Feb 2019 16:53:21 +0000 (UTC) Received: by mail-lj1-f175.google.com with SMTP id z25so14575952ljk.8 for ; Wed, 27 Feb 2019 08:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=V2nSA49/Bt5GTEKwqtepeORoxUh4CepfGbjC8SaGAgA=; b=qhhJhglns6vGmLN8+8gnKrLFdi7Q3yLmQrxnpQ3/Urt7/U8mJC2O6CzWzfUNifN7L7 oWGnRv2jlQhDacekW3BciJ4MwKWY8YDJDP6LPUMmGMSM990N3WYaEtW37n4AAQ/67fzn aIQbaGpQYFKm1cj/a22bDG82sKCUjxvsBxFLB3dQhPb264QjZQE0XnJiqLE+p66hF+ec WW0rivCkm7c0kFAhBwtkyCXh0TAjZHXr/hXFBpc/X5ldDDluRtgmSYUO77JYT+cafvo4 9v4JyC1uytyVP9pD5V///bxG30/9Y0KwJ+4clOhOLUlQsSpSQQFPZU9smVaIQyGXsJCS 89Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=V2nSA49/Bt5GTEKwqtepeORoxUh4CepfGbjC8SaGAgA=; b=lXyOPTvoPSMCkIYtmhSTzBdUP5nqZsxHI2Lpp65cx+tmIW6oVcGksLGQF1WY6SJ3aP j+VBxf0bm6irrlzbGcevzQ0XfNYNubv1pQC3/hdkVDEawbMB5BruAQiSH1hfXhVKwHMf Rjhc/jglLACdtcq26/Awb0QazgZbBTv59If3mpsO4tfugwC3mH903p+NpG8JCZ4Vdzic oYquJ+S3M0cJLPp3PVGkoV6unFay3pSxi97mDz+Lvs87OINWv75mpuOIFWgh4s/qfRwq jw8/xUo8R2x3svIAYeUCtTzJ+Fdo4c9DMMAons/qGEuSiaeFjLyjPmgvFMHwnlLQuTlR G62A== X-Gm-Message-State: APjAAAWSIe0+yERxdlsTK79kvZBwq9g/FzGqE7/4mJMXNndEf3xoPY7e DmunK0trhN1otzx0ZVW5Sg7SuW0k1pMPAa4erIzY0VEl X-Google-Smtp-Source: APXvYqypkpKviRQzPrPSPB7ppgjanfYjClMy1h6jM3U+jAZatH5L4wSN2RJiLi1ODNPFmA/CTC0sY66dwC/n21Mv9PU= X-Received: by 2002:a2e:63c3:: with SMTP id s64mr2163310lje.121.1551286395060; Wed, 27 Feb 2019 08:53:15 -0800 (PST) MIME-Version: 1.0 From: Maxim Muzafarov Date: Wed, 27 Feb 2019 19:53:03 +0300 Message-ID: Subject: [DISCUSSION] Hot cache backup To: dev@ignite.apache.org Content-Type: text/plain; charset="UTF-8" Igniters, Some of the stores with which the Apache Ignite is often compared has a feature called Snapshots [1] [2]. This feature provides an eventually consistent view on stored data for different purposes (e.g. moving data between environments, saving a backup of data for the further restore procedure and so on). The Apache Ignite has all opportunities and machinery to provide cache and\or data region snapshots out of the box but still don't have them. This issue derives from IEP-28 [5] on which I'm currently working on (partially described in the section [6]). I would like to solve this issue too and make Apache Ignite more attractive to use on a production environment. I've haven't investigated in-memory type caches yet, but for caches with enabled persistence, we can do it without any performance impact on cache operations (some additional IO operations are needed to copy cache data to backup store, copy on write technique is used here). We just need to use our DiscoverySpi, PME and Checkpointer process the right way. For the first step, we can store all backup data on each of cache affinity node locally. For instance, the `backup\snapshotId\cache0` folder will be created and all `cache0` partitions will be stored there for each local node for the snapshot process with id `snapshotId`. In future, we can teach nodes to upload snapshotted partitions to the one remote node or cloud. -- High-level process overview A new snapshot process is managed via DiscoverySpi and CommunicationSpi messages. 1. The initiator sends a request to the cluster (DiscoveryMessage). 2. When the node receives a message it initiates PME. 3. The node begins checkpoint process (holding write lock a short time) 4. The node starts to track any write attempts to the snapshotting partition and places the copy of original pages to the temp file. 5. The node performs merge the partition file with the corresponding delta. 6. When the node finishes the backup process it sends ack message with saved partitions to the initiator (or the error response). 7. When all ack messages received the backup is finished. The only problem here is that when the request message arrives at the particular node during running checkpoint PME will be locked until it ends. This is not good. But hopefully, it will be fixed here [4]. -- Probable API From the cache perspective: IgniteFuture snapshotFut = ignite.cache("default") .shapshotter() .create("myShapshotId"); IgniteSnapshot cacheSnapshot = snapshotFut.get(); IgniteCache copiedCache = ignite.createCache("CopyCache") .withConfiguration(defaultCache.getConfiguration()) .loadFromSnapshot(cacheSnapshot.id()); From the command line perspective: control.sh --snapshot take cache0,cache1,cache2 -- WDYT? Will it be a useful feature for the Apache Ignite? [1] https://geode.apache.org/docs/guide/10/managing/cache_snapshots/chapter_overview.html [2] https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsBackupTakesSnapshot.html [3] http://apache-ignite-developers.2346864.n4.nabble.com/Data-Snapshots-in-Ignite-td4183.html [4] https://issues.apache.org/jira/browse/IGNITE-10508 [5] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing [6] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-Checkpointer