ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: MemoryPolicy size vs cache size
Date Thu, 03 Aug 2017 14:23:59 GMT
Hi Pascal,

Looks like you do not actually run a query. Ignite doesn't fetch full
resultset by default and use paging for fetching.

Try to make *cursor.getAll() *called on every iteration
or (which is better) get iterator and iterate over rows to reduce heap
memory pressure.


On Wed, Aug 2, 2017 at 2:07 PM, Pascal <pascal.schuchhard@gmail.com> wrote:

> Hi,
>
> I try to use Ignite 2.1 to simulate the situation where a table doesn't fit
> in memory and data is read from disk every time a full table scan occurs.
> I've adapted the PersistentStoreExample to create a cache of a million
> items
> which is stored in 200MB of data files and 120MB of index files. I call
> PersistentStoreConfiguration.setFileIOFactory() to register a custom
> FileIOFactory that tracks what file I/O is occurring. This tracker shows
> the
> files are read during startup of the example, but not while the SQL query
> is
> running. The cache shouldn't fit in 10MB and I would expect that during the
> query the data would be read from disk. Why is this not happening and what
> do I need to change to test this?
>
> Thanks for your help.
> Pascal
>
> <bean class="org.apache.ignite.configuration.MemoryConfiguration">
>         <property name="pageSize" value="2048"/>
>         <property name="systemCacheInitialSize" value="#{10 * 1024 *
> 1024}"/>
>         <property name="systemCacheMaxSize" value="#{10 * 1024 * 1024}"/>
>         <property name="defaultMemoryPolicyName" value="default_mem_plc"/>
>         <property name="memoryPolicies">
>                 <list>
>                         <bean class="org.apache.ignite.configuration.
> MemoryPolicyConfiguration">
>                                 <property name="name"
> value="default_mem_plc"/>
>                                 <property name="initialSize" value="#{10 *
> 1024 * 1024}"/>
>                                 <property name="maxSize" value="#{10 *
> 1024 * 1024}"/>
>                         </bean>
>                 </list>
>         </property>
> </bean>
>
> public class PersistentStoreExample {
>     private static final boolean UPDATE = false;
>     private static final long COUNT = 1000000;
>     public static void doMain() throws Exception {
>         Ignition.setClientMode(false);
>         Ignite ig =
> Ignition.start("D:/Ignite/2.1/examples/config/persistentstore/example-
> persistent-store.xml");
>         IgniteConfiguration ic = ig.configuration();
>         PersistentStoreConfiguration pc =
> ic.getPersistentStoreConfiguration();
>         pc.setPersistentStorePath("C:\\tmp\\");
>         MyFileIOFactory myFact = new MyFileIOFactory();
>         pc.setFileIOFactory(myFact);
>         ig.active(true);
>         IgniteCache<Long, MyPerson> cache = ig.cache("personCache");
>
>         if (UPDATE) {
>             System.out.println("Populating the cache...");
>             try (IgniteDataStreamer<Long, MyPerson> streamer =
> ig.dataStreamer("personCache")) {
>                 streamer.allowOverwrite(true);
>                 for (long i = 0; i < COUNT; i++) {
>                     MyPerson o = new MyPerson(i, "per-" + i);
>                     streamer.addData(i, o);
>                     if (i > 0 && i % 10_000 == 0) System.out.println("Done:
> " + i);
>                 }
>             }
>         }
>
>         while(true)
>         {
>             Thread.sleep(3000);
>             System.out.println("\nFull scan " + COUNT + " times...");
>             for(int getId = 0; getId< COUNT; getId++) {
>                 QueryCursor cur = cache.query(new SqlFieldsQuery("select
> id,
> name from MyPerson where name like ?").setArgs("per-" + getId));
>                 if(getId % 100000 == 0) {
>                     System.out.println("  SQL Result: " + cur.getAll());
>                 }
>             }
>         }
>     }
> }
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/MemoryPolicy-size-vs-cache-size-tp15892.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Best regards,
Andrey V. Mashenkov

Mime
View raw message