ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Data eviction/expiration from Ignite persistence
Date Tue, 21 Nov 2017 17:45:43 GMT
We might break the compatibility for the next major release or even create a tool that will
migrate persistence files from an old to new formats.

—
Denis

> On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov <dpavlov.spb@gmail.com> wrote:
> 
> Hi Denis,
> 
> Second fix we need to do is B+ tree separation in per-partition basis:
> https://issues.apache.org/jira/browse/IGNITE-5874
> 
> Should we take into account compatibilty issues with previous Ignite
> persistent store versions, because current TTL tree is persisted, and will
> change its format?
> 
> Sincerely,
> Dmitriy Pavlov
> 
> 
> вт, 21 нояб. 2017 г. в 2:13, Denis Magda <dmagda@apache.org>:
> 
>> Dmitriy,
>> 
>> That’s about TTL and eviction support for Ignite persistence. Presently if
>> you set an expiration or eviction policy for a cache it will be applied for
>> data stored in memory. The policy never affects the persistence layer.
>> 
>> —
>> Denis
>> 
>>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>> wrote:
>>> 
>>> Hi Denis,
>>> 
>>> Is this need covered by PDS + TTL?
>>> 
>>> For the very first TTL test, I found some delay after applying TTL with
>> the
>>> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
>>> 
>>> And I'm wondering if the user's needs are covered by
>>> https://apacheignite.readme.io/docs/expiry-policies plus
>>> https://apacheignite.readme.io/docs/distributed-persistent-store
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <dpavlov.spb@gmail.com>:
>>> 
>>>> Hi Denis,
>>>> 
>>>> What is the difference of required by users functionality with TTL cache
>>>> expiration?
>>>> 
>>>> By some posts I can suppose TTL cache is compatible with native
>>>> persistence.
>>>> 
>>>> Sincerely,
>>>> Dmitriy Pavlov
>>>> 
>>>> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <dmagda@apache.org>:
>>>> 
>>>>> Igniters,
>>>>> 
>>>>> I’ve been talking to many Ignite users here and there who are already
>> on
>>>>> Ignite persistence or consider to turn it on. The majority of them are
>> more
>>>>> than satisfied with its current state and provided capabilities.
>> That’s is
>>>>> really good news for us.
>>>>> 
>>>>> However, I tend to come across the people who ask about
>>>>> eviction/expiration policies for the persistence itself. Had around 6
>>>>> conversation about the topic this month only.
>>>>> 
>>>>> Usually the requirement is connected with a streaming use case. When
an
>>>>> application streams a lot of data (IoT, metrics, etc.) to the cluster
>> but
>>>>> the data becomes stale in some period of time (day, couple of days,
>> etc.).
>>>>> The user doesn’t want to waste the disk space and needs to simple
>> purge the
>>>>> data from there.
>>>>> 
>>>>> My suggestion here is to create a timer task that will remove the stale
>>>>> data from the cluster. However, since the demand is growing probably
>> it’s a
>>>>> good time to discuss a feasibility of this feature.
>>>>> 
>>>>> Alex G, as the main architect of the persistence, could you share your
>>>>> thoughts on this? What will it cost to us to support
>> eviction/expiration
>>>>> for the persistence?
>>>>> 
>>>>> —
>>>>> Denis
>>>> 
>>>> 
>> 
>> 


Mime
View raw message