From dev-return-36039-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Wed Jun 27 22:26:21 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1410118066B for ; Wed, 27 Jun 2018 22:26:20 +0200 (CEST) Received: (qmail 59972 invoked by uid 500); 27 Jun 2018 20:26:20 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 59824 invoked by uid 99); 27 Jun 2018 20:26:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jun 2018 20:26:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1158BC6C5B for ; Wed, 27 Jun 2018 20:26:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.869 X-Spam-Level: ** X-Spam-Status: No, score=2.869 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Cx6c4YWWHz-P for ; Wed, 27 Jun 2018 20:26:17 +0000 (UTC) Received: from mail-it0-f68.google.com (mail-it0-f68.google.com [209.85.214.68]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 391EC5F11F for ; Wed, 27 Jun 2018 20:26:16 +0000 (UTC) Received: by mail-it0-f68.google.com with SMTP id p17-v6so4341447itc.2 for ; Wed, 27 Jun 2018 13:26:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=oEWonu6IwLwkKf2z0GP45ri0J4iiHO83KDmYvDrrnqo=; b=ZRwMgvxCXZos8xBUyTpiQmb+J/8rF9s7cHLakooUNFsdDOR5pUB6siiqHOFAhN/cOu nZjgZQAQTRGVTXrtFztuUon4Glt917bEoIwH2+f6T322Lf5dhuTAZmoCccujnWaJcTtK mvpUmL7bURwPnD/lmObRiIpDEeFoYtq6w4ZkTRniabMHnubAZZlXMXOww15Pd+U64pPf wAT4NPL65UmmFl+NnK/6m18WUxpsHX8UC0mVI6JedC8pGHfngO6KKGqwBRQxeO5fiJK+ Go1iX2nwFlB2rBw4uRJBza6PnU96N6lwhiCvxIEyM9N0AxusGCWQZ5vdfMHb2d9Rvkaz ctoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=oEWonu6IwLwkKf2z0GP45ri0J4iiHO83KDmYvDrrnqo=; b=dJaJ/qibR7Ro4pxYsCZl2wDTkKv3iY/BB+IKaj73cxaaMS6sJXORrWEi75Zn2+Ewcz w27/s3azzxcAni05AUY5p+IYAZyqYu8Ic7ip/pW9SFZw7x96DmgtPUGxAE3MmxbMBeSh o+Bu2YDnhIW9R4IGcQy+nVS+rF4LgOouTHH9dL/XUmmLR9apks94o0A7APcy/ojoIWje 6J8c77X/Y5Qy86TAvVAD0Oa2GzbHO0E9Qatddtwhl2Ge1cQUWKaGaTH5KlVLvcU/fpLF I6p1Mi9OlrILBC8yw7AFp6vFDzH0mFEFE7oamsY7j7fgRQ0PHTUYr5/U5CCIasBHyrLq 8aRA== X-Gm-Message-State: APt69E10cRfLLoSK7ezRpt2M558IyRHHcFHl15y+vSXUnWtaK5OmYLwa C9fieMCMx7X30c7JEdKzcMYxb8kNavfjtdSmGck= X-Google-Smtp-Source: ADUXVKLkdrKCiEgF6cVCBR/N068AFPsYa7ibwHwH7I57nxnt35Ss+rzrl5LNGhwJudfLGxnyFTohF1bphc8gaaZXD/s= X-Received: by 2002:a24:3c8:: with SMTP id e191-v6mr6639091ite.70.1530131174616; Wed, 27 Jun 2018 13:26:14 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Amir Akhmedov Date: Wed, 27 Jun 2018 16:26:02 -0400 Message-ID: Subject: Re: IgniteSet implementation: changes required To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="0000000000001f74e4056fa56db3" --0000000000001f74e4056fa56db3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes, you are right. Thanks, Amir On Wed, Jun 27, 2018 at 1:15 PM Denis Magda wrote: > Got you. If it's about redundant data duplication in onheap region then n= o > any concerns from my side. > > Anyway, considering that the data structure will be interacting with the > page memory directly then its entries can be stored in Ignite persistence > automatically (if the latter is on). Does it mean that the data structure > will be fully recovered after a restart and its entries can be pulled fro= m > disk on demand? > > -- > Denis > > > On Tue, Jun 26, 2018 at 1:49 PM Amir Akhmedov > wrote: > > > I also think it will better to remove setDataMap support cause > > 1. It's making extra pressure on GC by keeping entries on heap > > 2. It has difficult logic to support with lots of nuances > > 3. To maintain setDataMap today GridCacheMapEntry calls > > cctx.dataStructures().onEntryUpdated() on each entry mutation. I think > it's > > unnecessary cohesion. > > 4. For the case with single Ignite cache for all collocated > datastructure, > > an iterator creation will not be much slower than current implementatio= n > > since we can run affinity call on the node where all entries reside. > Also, > > we can create a better affinity mapper to fairly distribute > datastructures > > across a cluster rather than mapping by datastructure's name. > > > > Thanks, > > Amir > > > > > > On Tue, Jun 26, 2018 at 8:10 AM Anton Vinogradov wrote: > > > > > Denis, > > > > > > I think that better case is to remove onheap optimisation/duplication= . > > > This brings no drop to frequently used operations (put/remove), but > even > > > will make it slightly faster. > > > > > > The only one question we have here is "is it possible to restore onhe= ap > > map > > > in easy way?". > > > Seems that answer is no, so, I vote for setDataMap removal. > > > > > > =D0=B2=D1=82, 26 =D0=B8=D1=8E=D0=BD. 2018 =D0=B3. =D0=B2 15:00, Denis= Magda : > > > > > > > Anton, > > > > > > > > Will it be possible to reuse such a functionality for the rest of > data > > > > structures? I would invest our time in this if all data structures > > would > > > be > > > > able to work with Ignite persistence this way. > > > > > > > > -- > > > > Denis > > > > > > > > On Tue, Jun 26, 2018 at 1:53 AM Anton Vinogradov > > wrote: > > > > > > > > > >> Why don't we read data straight from the persistence layer > warming > > > RAM > > > > > up > > > > > >> in the background? > > > > > Because it's not a trivial task to finish such loading on unstabl= e > > > > > topology. > > > > > That's possible, ofcourse, but solution and complexity will be > almost > > > > > equals to WAL enable/disable. > > > > > > > > > > =D0=BF=D0=BD, 25 =D0=B8=D1=8E=D0=BD. 2018 =D0=B3. =D0=B2 22:13, D= enis Magda : > > > > > > > > > > > Folks, > > > > > > > > > > > > Why don't we read data straight from the persistence layer > warming > > > RAM > > > > up > > > > > > in the background? (like we do for SQL and other APIs). If it's= a > > > > > question > > > > > > of time, then I would suggest us not to hurry up and do it in a > > right > > > > > way. > > > > > > > > > > > > -- > > > > > > Denis > > > > > > > > > > > > On Mon, Jun 25, 2018 at 6:20 AM Anton Vinogradov > > > > wrote: > > > > > > > > > > > > > +1 to removal in case there is no easy, fast and consistent w= ay > > to > > > > > > restore > > > > > > > setDataMap on node restart. > > > > > > > I see that we'll gain some performance drop on size() or > keys(), > > > but > > > > > > these > > > > > > > methods are rarely used. > > > > > > > > > > > > > > =D0=BF=D0=BD, 25 =D0=B8=D1=8E=D0=BD. 2018 =D0=B3. =D0=B2 16:0= 7, Pavel Pereslegin < > xxtern@gmail.com > > >: > > > > > > > > > > > > > > > Hello, Igniters. > > > > > > > > > > > > > > > > I tried to implement IgniteSet data recovery when persisten= ce > > > > enabled > > > > > > > > [1] using trivial cache scanning, however I cannot find > optimal > > > way > > > > > to > > > > > > > > do that because of the following reasons: > > > > > > > > - Performing operations on IgniteSet requires completion of > > data > > > > > > > > loading (restoring of setDataMap) on all nodes. Do this > during > > > > > > > > partition map exchange is too long. > > > > > > > > - The prohibition of operations on IgniteSet before the > > > completion > > > > of > > > > > > > > asynchronous cache scanning on all nodes looks rather > > > complicated, > > > > > > > > because It is necessary to support all situations of unstab= le > > > > > > > > topology. > > > > > > > > > > > > > > > > So I see one option to fix data loss on node restart - remo= ve > > the > > > > > > > > entire optimization (setDataMap) and rework the iterator > > > > > > > > implementation to perform cache scanning. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5553 > > > > > > > > > > > > > > > > > > > > > > > > 2018-03-17 8:20 GMT+03:00 Andrey Kuznetsov < > stkuzma@gmail.com > > >: > > > > > > > > > Thanks, Dmitry. I agree ultimately, DS API uniformity is = a > > > > weighty > > > > > > > > reason. > > > > > > > > > > > > > > > > > > 2018-03-17 3:54 GMT+03:00 Dmitriy Setrakyan < > > > > dsetrakyan@apache.org > > > > > >: > > > > > > > > > > > > > > > > > >> On Fri, Mar 16, 2018 at 7:39 AM, Andrey Kuznetsov < > > > > > > stkuzma@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > Dmitry, your way allows to reuse existing > {{Ignite.set()}} > > > API > > > > > to > > > > > > > > create > > > > > > > > >> > both set flavors. We can adopt it unless somebody in t= he > > > > > community > > > > > > > > >> objects. > > > > > > > > >> > Personally, I like {{IgniteCache.asSet()}} approach > > proposed > > > > by > > > > > > > > Vladimir > > > > > > > > >> O. > > > > > > > > >> > more, since it emphasizes the difference between sets > > being > > > > > > created, > > > > > > > > but > > > > > > > > >> > this will require API extension. > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> Andrey, I am suggesting that Ignite.set(...) in > > non-collocated > > > > > mode > > > > > > > > behaves > > > > > > > > >> exactly the same as the proposed IgniteCache.asSet() > > method. I > > > > do > > > > > > not > > > > > > > > like > > > > > > > > >> the IgniteCache.asSet() API because it is inconsistent > with > > > > Ignite > > > > > > > data > > > > > > > > >> structure design. All data structures are provided on > Ignite > > > API > > > > > > > > directly > > > > > > > > >> and we should not change that. > > > > > > > > >> > > > > > > > > >> D. > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Best regards, > > > > > > > > > Andrey Kuznetsov. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --0000000000001f74e4056fa56db3--