ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Eviction policies with persistence
Date Wed, 07 Feb 2018 23:31:24 GMT
Thanks folks. I’ll improve the doc by 2.4 release:
IGNITE-7645 - Clarify eviction policy documentation <https://issues.apache.org/jira/browse/IGNITE-7645>

—
Denis

> On Feb 6, 2018, at 11:25 PM, Alexey Goncharuk <alexey.goncharuk@gmail.com> wrote:
> 
> Denis,
> 
> Yes, if there are no concurrent updates and there are no ongoing
> transactions preventing the entries to be evicted, then the whole page will
> be purged.
> 
> 2018-02-07 0:23 GMT+03:00 Denis Magda <dmagda@apache.org>:
> 
>> Alexey,
>> 
>> Let me understand this part because it’s still not crystal clear to me:
>> 
>>> Now as per the
>>> eviction mechanism, it is both per-page and per-entry: first, we choose a
>>> page which fits most for eviction, however, we cannot simply erase the
>> page
>>> because quite a lot of data structures are referring to that page. To
>> deal
>>> with it, we read keys that the chosen page contains and then clear
>>> individual cache entries, which in turn will clear up all necessary
>> links.
>> 
>> 
>> However will all the keys-values from a chosen page be removed eventually
>> during the eviction phase? If it’s true then it’s still a page based
>> eviction - we select 5 random oldest pages and purge all the key-values
>> from them.
>> 
>> —
>> Denis
>> 
>>> On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk <
>> alexey.goncharuk@gmail.com> wrote:
>>> 
>>> Guys,
>>> 
>>> To clarify the questions, I would like to reiterate over the mechanics of
>>> evictions and then suggest a renaming that should resolve such things in
>>> the future.
>>> 
>>> First of all, Lucas is right - eviction policy only makes sense when
>> native
>>> persistence is disabled because what it does is actually freeing up
>> memory
>>> when a user hits the memory limit. The only way to do this is to destroy
>>> inserted data because there is no other way to free memory. Now as per
>> the
>>> eviction mechanism, it is both per-page and per-entry: first, we choose a
>>> page which fits most for eviction, however, we cannot simply erase the
>> page
>>> because quite a lot of data structures are referring to that page. To
>> deal
>>> with it, we read keys that the chosen page contains and then clear
>>> individual cache entries, which in turn will clear up all necessary
>> links.
>>> If there are no concurrent updates, the page becomes empty and will be
>>> reused for other user data. This is exactly what is explained on the wiki
>>> page (at least in my reading of the page).
>>> 
>>> Second, at this point, I would rename page management mechanism with
>>> enabled persistence from 'eviction' to 'page replacement' or 'page
>>> swapping', because it does not destroy any data, but rather replaces a
>>> chosen buffer in memory from one page to another. The content of neither
>>> pages does not change during page replacement. This mechanism is unlikely
>>> to be selected by a user because the effectiveness of page replacements
>>> heavily depends on internal data structures and may change from version
>> to
>>> version, and may even be adaptive depending on the load pattern.
>>> 
>>> Hope this resolves the confusion.
>>> 
>>> --AG
>>> 
>>> 2018-02-03 1:03 GMT+03:00 Denis Magda <dmagda@apache.org>:
>>> 
>>>> Dmitriy,
>>>> 
>>>> I’m surprised to hear that Random-LRU works at the entry level when
>> Ignite
>>>> persistence is off. Frankly, this section confuses a lot:
>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
>>>> underthehood-Entryeviction <https://cwiki.apache.org/
>>>> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
>>>> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
>>>> 
>>>> While it was assumed that the entry-based eviction works only for
>> on-heap
>>>> caches:
>>>> https://apacheignite.readme.io/docs/evictions <
>>>> https://apacheignite.readme.io/docs/evictions>
>>>> 
>>>> Alex G., please chime in and clarify the misunderstanding.
>>>> 
>>>> —
>>>> Denis
>>>> 
>>>>> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>> wrote:
>>>>> 
>>>>> Hi Val,
>>>>> 
>>>>> I think it is quite accurate because eviction in case PDS enabled or
>> not
>>>>> has quite different purposes.
>>>>> 
>>>>> 1. Let us consider PDS enabled and page eviction occurs. First of all
>> it
>>>> is
>>>>> page based eviction, but not entry-based. Actually data is not removed,
>>>> but
>>>>> only written to disk. We can address this page later by ID.
>>>>> PDS eviction is primarily the replacement of pages from memory to page
>> on
>>>>> disk. This eviction policy should ensure the maximum performance of the
>>>>> solution in the future. There is no data removal from grid in this
>> case.
>>>>> And Ignite does not allow users to configure the policy. If benchmarks
>>>> show
>>>>> that a change in policy results in increased performance, then we can
>>>>> switch policy.
>>>>> 
>>>>> 2. Entry-based eviction (if there is no persistence is available) is
>>>>> slithly different. User will need to reload data into grid in case
>> entry
>>>> is
>>>>> evicted but required in cache. In that case Ignite provides selection
>> of
>>>>> policies.
>>>>> 
>>>>> Sincerely,
>>>>> Dmirtiy Pavlov
>>>>> 
>>>>> 
>>>>> 
>>>>> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com>:
>>>>> 
>>>>>> Guys,
>>>>>> 
>>>>>> Thanks for responses. But I still fail to understand how it affects
a
>>>> user.
>>>>>> 
>>>>>> Are you saying that with persistence enabled Random-LRU is always
used
>>>>>> regardless of what is specified in pageEvictionMode, while in
>> in-memory
>>>>>> only scenario we can also utilize Random-2-LRU for eviction? Is this
>>>>>> accurate? Are there any other differences?
>>>>>> 
>>>>>> -Val
>>>>>> 
>>>>>> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi Vyacheslav,
>>>>>>> 
>>>>>>> Yes, this is right, but for now PDS page based eviction uses
>>>> Random-LRU,
>>>>>>> not Random-2-LRU. Something may be changed in future Ignite releases,
>>>> but
>>>>>>> now Random-LRU is used. I am going to research if there is some
room
>>>> for
>>>>>>> improvements in this aspect (e.g. IGNITE-7507 recenly found).
>>>>>>> 
>>>>>>> If clean page is evicted, nothing really happends. If dirty page
is
>>>>>> evicted
>>>>>>> from region during checkpointing process run, page is stored
in file
>>>>>> before
>>>>>>> eviction
>>>>>>> 
>>>>>>> Some info about this can be found in
>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
>>>>>>> underthehood-Pagebasedeviction
>>>>>>> 
>>>>>>> Sincerely,
>>>>>>> Dmitriy Pavlov
>>>>>>> 
>>>>>>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur <daradurvs@gmail.com
>>> :
>>>>>>> 
>>>>>>>> Hi Valentin,
>>>>>>>> 
>>>>>>>> As far as I know, when persistence is enabled and Ignite
can't
>>>>>>>> allocate new page-memory in the memory segment, then Ignite
>>>>>>>> automatically evict some data from RAM to PDS using Random-2-LRU
>>>>>>>> eviction algorithm. And there are no mechanisms to evict
entries out
>>>>>>>> when PDS is enabled.
>>>>>>>> 
>>>>>>>> I'll be glad if I am not right and someone will correct me
because,
>> as
>>>>>>>> Ignite user, I can't use PDS only as RAM disk replication
for the
>> use
>>>>>>>> case in my project.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
>>>>>>>> <valentin.kulichenko@gmail.com> wrote:
>>>>>>>>> Folks,
>>>>>>>>> 
>>>>>>>>> On "Eviction Policies" documentation page [1] we have
the following
>>>>>>>> callout:
>>>>>>>>> 
>>>>>>>>>> Configured eviction policy has no effect if Ignite
persistence is
>>>>>>>> enabled
>>>>>>>>>> Note that if Ignite Persistence is enabled, then
the page-based
>>>>>>>> evictions
>>>>>>>>> have no effect because the oldest pages will be purged
from memory
>>>>>>>>> automatically.
>>>>>>>>> 
>>>>>>>>> This really confuses me. Why is there a difference in
how data is
>>>>>>> evicted
>>>>>>>>> from memory depending on having disk enabled or not?
And how does
>>>>>>>> eviction
>>>>>>>>> work with persistence enabled then?
>>>>>>>>> 
>>>>>>>>> Does anyone can clarify?
>>>>>>>>> 
>>>>>>>>> [1] https://apacheignite.readme.io/docs/evictions
>>>>>>>>> 
>>>>>>>>> -Val
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Best Regards, Vyacheslav D.
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message