ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Cache expiry policy not deleting records from disk(native persistence)
Date Fri, 13 Sep 2019 16:46:35 GMT
Artem, thanks, could you please share a reference to the updated page?
Can't find anything here:
https://apacheignite.readme.io/docs/evictions

Shiva, I've restarted the discussion on the dev list, to get to the bottom
of this gap and how it can be addressed:
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html

-
Denis


On Fri, Sep 13, 2019 at 7:05 AM Artem Budnikov <a.budnikov.ignite@gmail.com>
wrote:

> Hi Denis,
>
> I updated the page about eviction policies. Not freeing up space on disk
> was not implemented for reasons explained in the dev-list thread. I'll
> update the page once more if/when a solution is implemented.
>
> Artem
> On 13.09.2019 00:34, Denis Magda wrote:
>
> Shiva,
>
> Hopefully, someone from the dev community will pick this ticket up soon
> and solve the task. In the meantime, Artem, would you mind documenting this
> limitation referring to ticket 10862?
>
> -
> Denis
>
>
> On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar <shivakumar.eng@gmail.com>
> wrote:
>
>> I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but
>> this is same as https://issues.apache.org/jira/browse/IGNITE-10862
>> Any idea on the timeline of these tickets?
>> In the documentation
>> https://apacheignite.readme.io/v2.7/docs/expiry-policies
>> it says when native persistence is enabled "*expired entries are removed
>> from both memory and disk tiers*" but in the disk it just mark the pages
>> as unwanted pages and same disk space used by these unwanted pages will be
>> used to store new pages but it will not remove unwanted pages from disk and
>> so it will not release disk space used by these unwanted pages.
>>
>> here is the developer's discussion link
>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>>
>>
>> On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar <shivakumar.eng@gmail.com>
>> wrote:
>>
>>> Hi
>>> I have deployed ignite on kubernetes and configured two seperate
>>> persistent volume for WAL and persistence.
>>> The issue Iam facing is same as
>>> https://issues.apache.org/jira/browse/IGNITE-10862
>>>
>>> Thanks
>>> Shiva
>>>
>>> On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <
>>> aealexsandrov@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I guess that generated WAL will take this disk space. Please read about
>>>> WAL here:
>>>>
>>>> https://apacheignite.readme.io/docs/write-ahead-log
>>>>
>>>> Please provide the size of every folder under /opt/ignite/persistence.
>>>>
>>>> BR,
>>>> Andrei
>>>> 9/6/2019 9:45 PM, Shiva Kumar пишет:
>>>>
>>>> Hi all,
>>>> I have set cache expiry policy like this
>>>>
>>>>
>>>>    </property>
>>>>    <property name="cacheConfiguration">
>>>>             <list>
>>>>                 <bean id="cache-template-bean" abstract="true"
>>>> class="org.apache.ignite.configuration.CacheConfiguration">
>>>>                   <property name="name" value="templateEternal*"/>
>>>>                   <property name="cacheMode" value="PARTITIONED"/>
>>>>                   <property name="backups" value="1"/>
>>>>                   <property name="groupName" value="groupEternal"/>
>>>>                   <property name="expiryPolicyFactory">
>>>>                     <bean
>>>> class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
>>>>                       <constructor-arg>
>>>>                         <bean class="javax.cache.expiry.Duration">
>>>>                           <constructor-arg value="MINUTES"/>
>>>>                           <constructor-arg value="10"/>
>>>>                         </bean>
>>>>                       </constructor-arg>
>>>>                     </bean>
>>>>                   </property>
>>>>
>>>>                 </bean>
>>>>             </list>
>>>>    </property>
>>>>
>>>>
>>>> And batch inserting records to one of the table which is created with
>>>> above cache template.
>>>> Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes
>>>> records started reducing(expiring) when I monitored from sqlline.
>>>>
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 248896
>>>> --------------------------------
>>>> 1 row selected (0.86 seconds)
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 222174
>>>> --------------------------------
>>>> 1 row selected (0.313 seconds)
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 118154
>>>> --------------------------------
>>>> 1 row selected (0.15 seconds)
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 76061
>>>> --------------------------------
>>>> 1 row selected (0.106 seconds)
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 41671
>>>> --------------------------------
>>>> 1 row selected (0.063 seconds)
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 18455
>>>> --------------------------------
>>>> 1 row selected (0.037 seconds)
>>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>>> DIMENSIONS;
>>>> --------------------------------
>>>>
>>>> COUNT(ID)
>>>> --------------------------------
>>>>
>>>> 0
>>>> --------------------------------
>>>> 1 row selected (0.014 seconds)
>>>>
>>>>
>>>> But in the meantime, the disk space used by the persistence store was
>>>> in the same usage level instead of decreasing.
>>>>
>>>>
>>>> [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h
>>>> /opt/ignite/persistence/; sleep 1s; done
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>>
>>>>
>>>>
>>>> This means that expiry policy not deleting records from the disk, but
>>>> ignite document says when expiry policy is set and native persistence is
>>>> enabled then it deletes records from disk as well.
>>>> Am I missing some configuration?
>>>> Any help is appreciated.
>>>>
>>>> Shiva
>>>>
>>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message