ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shiva Kumar <shivakumar....@gmail.com>
Subject Re: Cache expiry policy not deleting records from disk(native persistence)
Date Tue, 10 Sep 2019 07:49:51 GMT
I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but
this is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation
https://apacheignite.readme.io/v2.7/docs/expiry-policies
it says when native persistence is enabled "*expired entries are removed
from both memory and disk tiers*" but in the disk it just mark the pages as
unwanted pages and same disk space used by these unwanted pages will be
used to store new pages but it will not remove unwanted pages from disk and
so it will not release disk space used by these unwanted pages.

here is the developer's discussion link
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html


On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar <shivakumar.eng@gmail.com>
wrote:

> Hi
> I have deployed ignite on kubernetes and configured two seperate
> persistent volume for WAL and persistence.
> The issue Iam facing is same as
> https://issues.apache.org/jira/browse/IGNITE-10862
>
> Thanks
> Shiva
>
> On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <aealexsandrov@gmail.com>
> wrote:
>
>> Hello,
>>
>> I guess that generated WAL will take this disk space. Please read about
>> WAL here:
>>
>> https://apacheignite.readme.io/docs/write-ahead-log
>>
>> Please provide the size of every folder under /opt/ignite/persistence.
>>
>> BR,
>> Andrei
>> 9/6/2019 9:45 PM, Shiva Kumar пишет:
>>
>> Hi all,
>> I have set cache expiry policy like this
>>
>>
>>    </property>
>>    <property name="cacheConfiguration">
>>             <list>
>>                 <bean id="cache-template-bean" abstract="true"
>> class="org.apache.ignite.configuration.CacheConfiguration">
>>                   <property name="name" value="templateEternal*"/>
>>                   <property name="cacheMode" value="PARTITIONED"/>
>>                   <property name="backups" value="1"/>
>>                   <property name="groupName" value="groupEternal"/>
>>                   <property name="expiryPolicyFactory">
>>                     <bean class="javax.cache.expiry.CreatedExpiryPolicy"
>> factory-method="factoryOf">
>>                       <constructor-arg>
>>                         <bean class="javax.cache.expiry.Duration">
>>                           <constructor-arg value="MINUTES"/>
>>                           <constructor-arg value="10"/>
>>                         </bean>
>>                       </constructor-arg>
>>                     </bean>
>>                   </property>
>>
>>                 </bean>
>>             </list>
>>    </property>
>>
>>
>> And batch inserting records to one of the table which is created with
>> above cache template.
>> Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records
>> started reducing(expiring) when I monitored from sqlline.
>>
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 248896
>> --------------------------------
>> 1 row selected (0.86 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 222174
>> --------------------------------
>> 1 row selected (0.313 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 118154
>> --------------------------------
>> 1 row selected (0.15 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 76061
>> --------------------------------
>> 1 row selected (0.106 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 41671
>> --------------------------------
>> 1 row selected (0.063 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 18455
>> --------------------------------
>> 1 row selected (0.037 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> --------------------------------
>>
>> COUNT(ID)
>> --------------------------------
>>
>> 0
>> --------------------------------
>> 1 row selected (0.014 seconds)
>>
>>
>> But in the meantime, the disk space used by the persistence store was in
>> the same usage level instead of decreasing.
>>
>>
>> [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h
>> /opt/ignite/persistence/; sleep 1s; done
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>
>>
>>
>> This means that expiry policy not deleting records from the disk, but
>> ignite document says when expiry policy is set and native persistence is
>> enabled then it deletes records from disk as well.
>> Am I missing some configuration?
>> Any help is appreciated.
>>
>> Shiva
>>
>>

Mime
View raw message