ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: loadCache takes long time to complete with million rows
Date Tue, 26 Apr 2016 12:09:38 GMT
Hi,

The most obvious reason - insufficient heap. Even though you specified
OFFHEAP_TIERED mode, Ignite still generates some intermediate objects as
well as some long-lived ones. Please monitor your application through a
profiler to confirm that it has enough heap and GC pauses are not too long.

Vladimir.

On Tue, Apr 26, 2016 at 11:25 AM, arthi <Arthi.Kasturirangan.ap@nielsen.com>
wrote:

> Hi Team,
>
> I am loading a partitioned cache with 30Million rows using loadCache API
> from a persistence store.
> The data gets loaded into the cache, but the process takes a long time to
> complete.
>
> Here is the config -
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>                                         <property name="name"
> value="SHOP_ITEM_BITMAP_CACHE" />
>                                         <property name="eagerTtl"
> value="false"/>
>                                         <property name="copyOnRead"
> value="false"/>
>                                         <property name="atomicityMode"
> value="ATOMIC"/>
>                     <property name="backups" value="0"/>
>                                         <property name="memoryMode"
> value="OFFHEAP_TIERED"/>
>                                         <property name="offHeapMaxMemory"
> value="0"/>
>                                         <property name="swapEnabled"
> value="false"/>
>                                         <property name="cacheMode"
> value="PARTITIONED" />
>                                         <property name="affinity">
>                                                 <bean
>
> class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
>                                                         <property
> name="partitions" value="64"/>
>                                                 </bean>
>                                         </property>
>                                         <property name="cacheStoreFactory">
>                                                 <bean
> class="javax.cache.configuration.FactoryBuilder$SingletonFactory">
>                                                   <constructor-arg>
>                                                         <bean
>
> class="com.nielsen.poc.aggregation.ignite.datagrid.store.ShopItemBitmapStore">
>                                                         </bean>
>                                                   </constructor-arg>
>                                                 </bean>
>                                         </property>
>                                         <property name="readThrough"
> value="true" />
>                                         <property name="queryEntities">
>                                                 <list>
>                                                         <bean
> class="org.apache.ignite.cache.QueryEntity">
>                                                                 <property
> name="keyType" value="org.apache.ignite.cache.AffinityKey"
> />
>                                                                 <property
> name="valueType"
> value="com.nielsen.poc.aggregation.ignite.datagrid.model.ShopItemBitmap" />
>                                                                 <property
> name="fields">
>
> <map>
>
>       <entry key="id" value="java.lang.Integer" />
>
>       <entry key="sid_per_id" value="java.lang.Integer" />
>
>       <entry key="sid_mah_id" value="java.lang.Integer" />
>
>       <entry key="sid_itm_id" value="java.lang.Integer" />
>
>       <entry key="sid_prm_id" value="java.lang.Integer" />
>
>       <entry key="sid_cha_code" value="java.lang.String" />
>
>       <entry key="sid_service" value="java.lang.String" />
>
>       <entry key="sid_itm_dist" value="java.lang.String" />
>
>       <entry key="category" value="java.lang.String" />
>
> </map>
>                                                                 </property>
>                                                                 <property
> name="indexes">
>
> <list>
>
>       <bean class="org.apache.ignite.cache.QueryIndex">
>
>               <constructor-arg index="0">
>
>                       <list>
>
>                               <value>sid_mah_id</value>
>
>                               <value>category</value>
>
>                               <value>sid_per_id</value>
>
>                       </list>
>
>               </constructor-arg>
>
>               <constructor-arg index="1" value="SORTED"/>
>
>       </bean>
>
> </list>
>                                                                 </property>
>                                                         </bean>
>                                                 </list>
>                                         </property>
>                                 </bean>
>
> can you please guide to see what the issue is?
> The same loadCache API can work faster for smaller data sets.
>
> Thanks,
> Arthi
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/loadCache-takes-long-time-to-complete-with-million-rows-tp4534.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Mime
View raw message