ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <nadav.h...@docauthority.com>
Subject RE: Apache Ignite starts fast and then become really slow with out of memory
Date Tue, 19 Feb 2019 18:26:06 GMT
Hey Ilya,

 

I managed to stable the memory and avoid extreme slowness by send compute jobs one by one
and not in bulk.

The compute jobs takes objects from cache by bulks, so I guess that firing 50 compute jobs
on a node where each job takes bulk of 100 is too heavy for it to handle…

My node has 32GB (Can go even higher) and 16 Cores.

 

The one by one bits the purpose of scaling (i.e. multiple nodes serving the same request),
 Is there a way to send a bulk of compute jobs and manage it  won't "explode"

 

Thank you very much!!

 

Nadav

 

From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com> 
Sent: Monday, February 18, 2019 11:39 AM
To: user@ignite.apache.org
Subject: Re: Apache Ignite starts fast and then become really slow with out of memory

 

Hello!

 

I recommend starting from simplest configuration to more complex one. Remove onheap cache,
see if problem goes away.

 

Regards,

-- 

Ilya Kasnacheev

 

 

вс, 17 февр. 2019 г. в 12:09, <nadav.hoze@docauthority.com <mailto:nadav.hoze@docauthority.com>
>:

Hey Ilya,

 

Yes you are right, the LinkedHashMaps entries are the dominate part. 

I was concentrating on the reader because I thought there may be an implicit connection..

Those LinkedHashMaps entries  are part of the object structure I'm loading to cache, if needed
I can send the model code (Java).

 

As you can see from my config, I use onheapCacheEnabled=true, my objects can be pretty heavy
(2 MG each), but I've place a rigid eviction policy of only 1000 entries. 

 

<property name="evictionPolicy">

<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">

<property name="maxSize" value="1000"/>

             </bean>

</property>

 

My heap is 16GB (can be more..)

 

if I set onheapCacheEnabled=false does it mean that the object loaded will not reside in the
heap..?

 

My general assumption is that I have problem with my GC tuning and it does not handle the
cache load well and struggles to clear/allocate memory.

 

What will help you to identify my problem more precisely ? (GC Logs, behavioral code, model
code) ?

 

Thank you  very much.

 

Nadav

 

From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com <mailto:ilya.kasnacheev@gmail.com>
> 
Sent: Friday, February 15, 2019 4:33 PM
To: user@ignite.apache.org <mailto:user@ignite.apache.org> 
Subject: Re: Apache Ignite starts fast and then become really slow with out of memory

 

Hello!

 

As far as I can see, the highlighted class does not dominate heap in any meaningful way.

 

You seem to have huge LinkedHashMaps, any idea where they are used/held?

 

Regards,

 

-- 

Ilya Kasnacheev

 

 

чт, 14 февр. 2019 г. в 22:11, <nadav.hoze@docauthority.com <mailto:nadav.hoze@docauthority.com>
>:

Hey,

 

Guys is this normal…?

 



 

So many binary readers are opened without reuse? I saw an improvement issue on this https://jira.apache.org/jira/browse/IGNITE-5721

 

I'm running a simple compute tasks that split around 4K ids to compute job that each one takes
from cache bulk of 500 ids. 

Each Object size return is ~1.8MB. 

 

When I run 3-4 iterations it is super fasts completing the tasks under 100 ms.

But then memory rises and it becomes really slow, fully occupied by GC and after a while it
caves and get OOME. 

 

I'm evaluating ignite for a POC and this is a real showstopper.

I'm sure that I'm missing something because I do nothing out of the ordinary.

 

I tried different GC tuning G1 and CMS, different evection policies,  different region sizes.
Nothing helped.

 

I'm using the latest ignite 2.7, my machine is windows server 12 R2 with 8 cores and 32GB
RAM, this is my node config:

 

<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xmlns:context="http://www.springframework.org/schema/context"

       xsi:schemaLocation="

        http://www.springframework.org/schema/beans

        http://www.springframework.org/schema/beans/spring-beans.xsd

        http://www.springframework.org/schema/context

           http://www.springframework.org/schema/context/spring-context.xsd

">

 

    <context:annotation-config/>

 

    <context:property-placeholder location="classpath:application.properties"/>

 

    <bean id="similarityConfiguration" class="com.cla.filecluster.service.files.pv.SimilarityConfiguration"/>

 

    <bean id="similarityCalculator" class="com.cla.filecluster.service.files.pv.SimilarityCalculator"

          factory-method="create">

        <constructor-arg ref="similarityConfiguration" />

    </bean>

                

                

 

    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

                  <property name="userAttributes">

            <map key-type="java.lang.String" value-type="java.lang.Boolean">

                <entry key="data.node" value="true"/>

            </map>

        </property>

                                

                                <property name="binaryConfiguration">

                                                <bean class="org.apache.ignite.configuration.BinaryConfiguration">

                                                                <property name="compactFooter"
value="false"/>

                                                </bean>

                                </property>

                                

                                <!-- property name="asyncCallbackPoolSize" value="2"/ -->

                                

                

                                

                                  <property name="dataStorageConfiguration">

            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">

                                                  <property name="pageSize" value="#{4
* 1024}"/>

                                                   

                                                                 <property name="defaultDataRegionConfiguration">

                                                                                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">

                                                                                         
      <property name="name" value="analysis_region"/>

                        <property name="persistenceEnabled" value="true"/>

                                                                                         
      <property name="initialSize" value="#{100 * 1024 * 1024}"/>

                                                                                         
      <property name="maxSize" value="8589934592"/>

                    </bean>

                                                                </property>

                <property name="storagePath" value="E:\asm-repo\db"/>

                <property name="walPath" value="E:\asm-repo\wal"/>

                                                                <property name="walArchivePath"
value="E:\asm-repo\wal\archive"/>

            </bean>

        </property>

 

        <property name="cacheConfiguration">

            <list>

                <!-- Partitioned cache example configuration (Atomic mode). -->

                <bean class="org.apache.ignite.configuration.CacheConfiguration">

                                                                                <property
name="storeKeepBinary" value="true"/>

                                                                                <property
name="eagerTtl" value="true"/>

                    <property name="name" value="PvAnalysisData"/>

                                                                                <property
name="dataRegionName" value="analysis_region"/>

                    <property name="atomicityMode" value="ATOMIC"/>

                    <property name="backups" value="0"/>

                                                                                <property
name="onheapCacheEnabled" value="true"/>

                                                                   <property name="expiryPolicyFactory">

                                                                                         
      <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">

                                                                                         
                      <constructor-arg>

                                                                                         
                                      <bean class="javax.cache.expiry.Duration">

                                                                                         
                                                      <constructor-arg value="SECONDS"/>

                                                                                         
                                                      <constructor-arg value="5"/>

                                                                                         
                                      </bean>

                                                                                         
                      </constructor-arg>

                                                                                         
      </bean>

                                                                                </property>

                                                                  <property name="evictionPolicy">

                                                                                

                                                                                <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">

                                                                                  <!--
property name="maxMemorySize" value="104857600"/ -->

                                                                                  <property
name="maxSize" value="1000"/>

                                                                                </bean>

                                                                  </property>

                                                                                         
      <property name="nodeFilter">

                                                                                         
                      <bean class="com.cla.filecluster.common.DataNodeFilter"/>

                                                                                         
      </property>

                </bean>

            </list>

        </property>

 

        <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->

        <property name="discoverySpi">

            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                <property name="ipFinder">

                    <!--

                        Ignite provides several options for automatic discovery that can be
used

                        instead os static IP based discovery. For information on all options
refer

                        to our documentation: http://apacheignite.readme.io/docs/cluster-config

                    -->

                    <!-- Uncomment static IP finder to enable static-based discovery of
initial nodes. -->

                    <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->

                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">

                        <property name="addresses">

                            <list>

                                <!-- In distributed environment, replace with actual host
IP address. -->

                                <value>127.0.0.1:47500..47509</value>

                            </list>

                        </property>

                    </bean>

                </property>

            </bean>

        </property>

    </bean>

</beans>

 

 

 

Thanks,


Nadav

System Architect



+972-544821606
 <http://www.docauthority.com/> DocAuthority.com

 

 

 


Mime
View raw message