From dev-return-30644-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Feb 2 20:36:24 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id BB513180608 for ; Fri, 2 Feb 2018 20:36:24 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id AB148160C49; Fri, 2 Feb 2018 19:36:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C8498160C25 for ; Fri, 2 Feb 2018 20:36:23 +0100 (CET) Received: (qmail 61519 invoked by uid 500); 2 Feb 2018 19:36:22 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 61505 invoked by uid 99); 2 Feb 2018 19:36:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Feb 2018 19:36:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id AE26C198C46 for ; Fri, 2 Feb 2018 19:36:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.998 X-Spam-Level: * X-Spam-Status: No, score=1.998 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gridgain-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id HGRi2vbYo-AS for ; Fri, 2 Feb 2018 19:36:19 +0000 (UTC) Received: from mail-pg0-f48.google.com (mail-pg0-f48.google.com [74.125.83.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 3AF055F1EE for ; Fri, 2 Feb 2018 19:36:18 +0000 (UTC) Received: by mail-pg0-f48.google.com with SMTP id w17so14507190pgv.6 for ; Fri, 02 Feb 2018 11:36:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gridgain-com.20150623.gappssmtp.com; s=20150623; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=nrJW794mJAjMHYR8kR6J9YqwQlrwc5zMRRe6G3MOkqk=; b=Fs+Wcl45RRtNVEpkhdcsYob/knH4OIHU8XTpjfuFPyh952Q9sH1JFYvf0GkL4s8j0O 3GFnVLzqMcH9/oihO6GpghQksfqVILcwH3BK9KENozn0hco/EU1Sc/vEWNOIbZ9LgZxt XXBF2oVsmo/s8m70y7vFgl2fepvz/hgnLGmcjFSv9zZTxb1qZi4XFeVIZkLP7QS0xMQ/ MKfPQOJE92SjzkOrU2VerZLlP9lgFuptgPwbnooebmGOkcF76wPwu/oxX8nGMqfsM0UX 39Uh0iSIYTgDQxOmLUXhhiYaIBgeYPXD4LVR2nJiCNOAnsbNTFz4xObyx3vniEOPV8cN BAUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=nrJW794mJAjMHYR8kR6J9YqwQlrwc5zMRRe6G3MOkqk=; b=d77+WSAlz07MIqpobryDm4M3iuyCInH6u5HEPr7ePM7EUyFABB8Qle4EArdl+nsQwM WxfOXXDtnG4YoKwLY3jr/SfIPNbevpNQBf7FxQCTK2mWsiT9Z8BqEau3Pn4XS+Bg465r zeC5ePUp+RiwVgfR/GH26xNdtGWbc9g5lcTAv4nbKm7nV48QqdctZXT5nKeG0lrTd3ue 7RYWLhVjAY/XTn8SX24dES3lsiuI6oUYXNLW4q+vH6cH7YB8vmx+RgzFUK+ypIqm1bL5 xlsJcC211qIeh1x9fU5gU5y88wmi9ZqQUw7/CK8TsTXJJuqgvb8JB/2ywdqOGZoP39XF VXjQ== X-Gm-Message-State: AKwxytdq72r0drPrNIVY44zmTAJdHBH1AcivKgVbelPosSEOUbrEvyIu wpAY4cI+3GHkmBLVtDRi+Vg2ypd8C8A= X-Google-Smtp-Source: AH8x225SnUIy+xF/yLXZq5UT+7mcZVo673M11VPpYFwYG+NCb442r+kBmbRaCcX1/C0eWXf97zq+xg== X-Received: by 10.98.192.11 with SMTP id x11mr546154pff.27.1517600171027; Fri, 02 Feb 2018 11:36:11 -0800 (PST) Received: from ?IPv6:2601:641:4000:3fad:49ad:c628:b9db:a428? ([2601:641:4000:3fad:49ad:c628:b9db:a428]) by smtp.gmail.com with ESMTPSA id p14sm4862458pff.108.2018.02.02.11.36.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Feb 2018 11:36:09 -0800 (PST) From: Lucas Beeler Content-Type: multipart/alternative; boundary="Apple-Mail=_004F4613-8620-421F-8ACE-5B810049CF43" Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: Eviction policies with persistence Date: Fri, 2 Feb 2018 11:36:08 -0800 References: To: dev@ignite.apache.org In-Reply-To: Message-Id: <2003DAE7-8D3A-466C-AC75-C147CE9F3C3B@gridgain.com> X-Mailer: Apple Mail (2.3445.5.20) --Apple-Mail=_004F4613-8620-421F-8ACE-5B810049CF43 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Guys, I agree with both Val and Dmitriy. Dmitriy is technically correct, but = Val is also right that we should clean up the language. Here=E2=80=99s what I mean by "cleaning up the language": unless I=E2=80=99= m missing something, eviction policy has no effect when Ignite = persistence is enabled, because Ignite will adaptively page-in and = page-out memory to disk in a fashion very similar to how modern = operating systems handle virtual memory. Using an explicitly configured = eviction policy really only matters if you=E2=80=99re using 3rd party = persistence (such as write-behind to an RDBMS). So basically, there are = two different use cases: Case #1: Native Persistence is On: eviction policy is meaningless = because paging is handled automatically by Ignite. The actual algorithm = used by Ignite doesn=E2=80=99t matter=E2=80=94so long as it = works=E2=80=94because in this case Ignite is a magical black box that = handles paging for you and you can=E2=80=99t configure it. Case #2: Native Persistence is Off: in this case, users have to decide = how to deal with out-of-memory situations by specifying their own = eviction policy. Ignite will not magically page to disk and so users = need to configure eviction. When configuring eviction, they have several = configuration options. So, in my mind, that documentation page needs to have a very clear = sentence in its first paragraph that says something like =E2=80=9Clook, = all of the stuff on this page only matters if you don=E2=80=99t have = persistence enabled.=E2=80=9D Just my two cents=E2=80=A6 Take care, Lucas -- Lucas BEELER Technical Consultant, Professional Services GridGain Systems www.gridgain.com > On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov = wrote: >=20 > Hi Val, >=20 > I think it is quite accurate because eviction in case PDS enabled or = not > has quite different purposes. >=20 > 1. Let us consider PDS enabled and page eviction occurs. First of all = it is > page based eviction, but not entry-based. Actually data is not = removed, but > only written to disk. We can address this page later by ID. > PDS eviction is primarily the replacement of pages from memory to page = on > disk. This eviction policy should ensure the maximum performance of = the > solution in the future. There is no data removal from grid in this = case. > And Ignite does not allow users to configure the policy. If benchmarks = show > that a change in policy results in increased performance, then we can > switch policy. >=20 > 2. Entry-based eviction (if there is no persistence is available) is > slithly different. User will need to reload data into grid in case = entry is > evicted but required in cache. In that case Ignite provides selection = of > policies. >=20 > Sincerely, > Dmirtiy Pavlov >=20 >=20 >=20 > =D1=87=D1=82, 1 =D1=84=D0=B5=D0=B2=D1=80. 2018 =D0=B3. =D0=B2 22:24, = Valentin Kulichenko < > valentin.kulichenko@gmail.com>: >=20 >> Guys, >>=20 >> Thanks for responses. But I still fail to understand how it affects a = user. >>=20 >> Are you saying that with persistence enabled Random-LRU is always = used >> regardless of what is specified in pageEvictionMode, while in = in-memory >> only scenario we can also utilize Random-2-LRU for eviction? Is this >> accurate? Are there any other differences? >>=20 >> -Val >>=20 >> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov >> wrote: >>=20 >>> Hi Vyacheslav, >>>=20 >>> Yes, this is right, but for now PDS page based eviction uses = Random-LRU, >>> not Random-2-LRU. Something may be changed in future Ignite = releases, but >>> now Random-LRU is used. I am going to research if there is some room = for >>> improvements in this aspect (e.g. IGNITE-7507 recenly found). >>>=20 >>> If clean page is evicted, nothing really happends. If dirty page is >> evicted >>> from region during checkpointing process run, page is stored in file >> before >>> eviction >>>=20 >>> Some info about this can be found in >>> https://cwiki.apache.org/confluence/display/IGNITE/ >>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory- >>> underthehood-Pagebasedeviction >>>=20 >>> Sincerely, >>> Dmitriy Pavlov >>>=20 >>> =D1=87=D1=82, 1 =D1=84=D0=B5=D0=B2=D1=80. 2018 =D0=B3. =D0=B2 9:53, = Vyacheslav Daradur : >>>=20 >>>> Hi Valentin, >>>>=20 >>>> As far as I know, when persistence is enabled and Ignite can't >>>> allocate new page-memory in the memory segment, then Ignite >>>> automatically evict some data from RAM to PDS using Random-2-LRU >>>> eviction algorithm. And there are no mechanisms to evict entries = out >>>> when PDS is enabled. >>>>=20 >>>> I'll be glad if I am not right and someone will correct me because, = as >>>> Ignite user, I can't use PDS only as RAM disk replication for the = use >>>> case in my project. >>>>=20 >>>>=20 >>>> On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko >>>> wrote: >>>>> Folks, >>>>>=20 >>>>> On "Eviction Policies" documentation page [1] we have the = following >>>> callout: >>>>>=20 >>>>>> Configured eviction policy has no effect if Ignite persistence is >>>> enabled >>>>>> Note that if Ignite Persistence is enabled, then the page-based >>>> evictions >>>>> have no effect because the oldest pages will be purged from memory >>>>> automatically. >>>>>=20 >>>>> This really confuses me. Why is there a difference in how data is >>> evicted >>>>> from memory depending on having disk enabled or not? And how does >>>> eviction >>>>> work with persistence enabled then? >>>>>=20 >>>>> Does anyone can clarify? >>>>>=20 >>>>> [1] https://apacheignite.readme.io/docs/evictions >>>>>=20 >>>>> -Val >>>>=20 >>>>=20 >>>>=20 >>>> -- >>>> Best Regards, Vyacheslav D. >>>>=20 >>>=20 >>=20 --Apple-Mail=_004F4613-8620-421F-8ACE-5B810049CF43--