ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Budnikov <a.budnikov.ign...@gmail.com>
Subject Re: Cache expiry policy not deleting records from disk(native persistence)
Date Fri, 13 Sep 2019 14:05:49 GMT
Hi Denis,

I updated the page about eviction policies. Not freeing up space on disk 
was not implemented for reasons explained in the dev-list thread. I'll 
update the page once more if/when a solution is implemented.

Artem

On 13.09.2019 00:34, Denis Magda wrote:
> Shiva,
>
> Hopefully, someone from the dev community will pick this ticket up 
> soon and solve the task. In the meantime, Artem, would you mind 
> documenting this limitation referring to ticket 10862?
>
> -
> Denis
>
>
> On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar <shivakumar.eng@gmail.com 
> <mailto:shivakumar.eng@gmail.com>> wrote:
>
>     I have filed a bug
>     https://issues.apache.org/jira/browse/IGNITE-12152 but this is
>     same as https://issues.apache.org/jira/browse/IGNITE-10862
>     Any idea on the timeline of these tickets?
>     In the documentation
>     https://apacheignite.readme.io/v2.7/docs/expiry-policies
>     it says when native persistence is enabled "*expired entries are
>     removed from both memory and disk tiers*" but in the disk it just
>     mark the pages as unwanted pages and same disk space used by these
>     unwanted pages will be used to store new pages but it will not
>     remove unwanted pages from disk and so it will not release disk
>     space used by these unwanted pages.
>
>     here is the developer's discussion link
>     http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>
>
>     On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar
>     <shivakumar.eng@gmail.com <mailto:shivakumar.eng@gmail.com>> wrote:
>
>         Hi
>         I have deployed ignite on kubernetes and configured two
>         seperate persistent volume for WAL and persistence.
>         The issue Iam facing is same as
>         https://issues.apache.org/jira/browse/IGNITE-10862
>
>         Thanks
>         Shiva
>
>         On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov,
>         <aealexsandrov@gmail.com <mailto:aealexsandrov@gmail.com>> wrote:
>
>             Hello,
>
>             I guess that generated WAL will take this disk space.
>             Please read about WAL here:
>
>             https://apacheignite.readme.io/docs/write-ahead-log
>
>             Please provide the size of every folder under
>             /opt/ignite/persistence.
>
>             BR,
>             Andrei
>
>             9/6/2019 9:45 PM, Shiva Kumar пишет:
>>             Hi all,
>>             I have set cache expiry policy like this
>>
>>
>>              </property>
>>              <property name="cacheConfiguration">
>>                       <list>
>>                           <bean id="cache-template-bean"
>>             abstract="true"
>>             class="org.apache.ignite.configuration.CacheConfiguration">
>>                             <property name="name"
>>             value="templateEternal*"/>
>>                             <property name="cacheMode"
>>             value="PARTITIONED"/>
>>                             <property name="backups" value="1"/>
>>                             <property name="groupName"
>>             value="groupEternal"/>
>>                             <property name="expiryPolicyFactory">
>>                               <bean
>>             class="javax.cache.expiry.CreatedExpiryPolicy"
>>             factory-method="factoryOf">
>>                                 <constructor-arg>
>>                                   <bean
>>             class="javax.cache.expiry.Duration">
>>                                     <constructor-arg value="MINUTES"/>
>>                                     <constructor-arg value="10"/>
>>                                   </bean>
>>                                 </constructor-arg>
>>                               </bean>
>>                             </property>
>>
>>                           </bean>
>>                       </list>
>>              </property>
>>
>>
>>             And batch inserting records to one of the table which is
>>             created with above cache template.
>>             Around 10 minutes, I ingested ~1.5GB of data and after 10
>>             minutes records started reducing(expiring) when I
>>             monitored from sqlline.
>>
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             248896
>>             --------------------------------
>>             1 row selected (0.86 seconds)
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             222174
>>             --------------------------------
>>             1 row selected (0.313 seconds)
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             118154
>>             --------------------------------
>>             1 row selected (0.15 seconds)
>>             0: jdbc:ignite:thin://192.168.*.*:10800>
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             76061
>>             --------------------------------
>>             1 row selected (0.106 seconds)
>>             0: jdbc:ignite:thin://192.168.*.*:10800>
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             41671
>>             --------------------------------
>>             1 row selected (0.063 seconds)
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             18455
>>             --------------------------------
>>             1 row selected (0.037 seconds)
>>             0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
>>             from DIMENSIONS;
>>             --------------------------------
>>
>>             COUNT(ID)
>>             --------------------------------
>>
>>             0
>>             --------------------------------
>>             1 row selected (0.014 seconds)
>>
>>
>>             But in the meantime, the disk space used by the
>>             persistence store was in the same usage level instead of
>>             decreasing.
>>
>>
>>             [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ;
>>             do df -h /opt/ignite/persistence/; sleep 1s; done
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>             Filesystem Size Used Avail Use% Mounted on
>>             /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>
>>
>>
>>             This means that expiry policy not deleting records from
>>             the disk, but ignite document says when expiry policy is
>>             set and native persistence is enabled then it deletes
>>             records from disk as well.
>>             Am I missing some configuration?
>>             Any help is appreciated.
>>
>>             Shiva
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message