ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "aaron@tophold.com" <aa...@tophold.com>
Subject In which scenario the ignite nodes will shutdown by itself
Date Tue, 15 Aug 2017 05:14:40 GMT
Hi All, 

we notice more than one time the ignite node and cache just stop itself quietly . this is
weird, no specific ERROR log print. 

we allocated enough memory for those cache.  From the GC log we can not find any obvious abnormal
as the main process still running . 

<property name="cacheMode" value="REPLICATED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="readFromBackup" value="true"/>


<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindBatchSize" value="8192"/>
<property name="writeBehindFlushThreadCount" value="5"/>
<property name="writeBehindFlushFrequency" value="10000"/>

<property name="storeKeepBinary" value="false"/>
<property name="onheapCacheEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
<property name="memoryPolicyName" value="Trans_Region"/>


<property name="evictionPolicy">
    <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
        <property name="maxSize" value="#{80000}"/>
    </bean>
</property>
we have enough memory for this cache:
<bean class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
    <property name="name" value="Trans_Region"/>
    <!--  8~12g -->
    <property name="initialSize" value="#{8L *1024L * 1024L * 1024L}"/>
    <property name="maxSize" value="#{12L *1024L * 1024L * 1024L}"/>
    <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
</bean>

the server have 64G memory totally 

But if we keep run for more than 5 hours then we keep getting:

javax.cache.CacheException: class org.apache.ignite.internal.NodeStoppingException: Operation
has been cancelled (node is stopping).
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1312)
~[ignite-core-2.1.0.jar!/:2.1.0]
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2630)
~[ignite-core-2.1.0.jar!/:2.1.0]
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1509)
~[ignite-core-2.1.0.jar!/:2.1.0]
        at com.tophold.trade.booker.TransactionCommandGateWay.handle(TransactionCommandGateWay.java:343)
~[classes!/:?]
        at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]

Eventually:

java.lang.IllegalStateException: class org.apache.ignite.internal.processors.cache.CacheStoppedException:
Failed to perform cache operation (cache is stopped): TransactionEntry
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.checkState(GridCacheGateway.java:88)
~[ignite-core-2.1.0.jar!/:2.1.0]
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:179)
~[ignite-core-2.1.0.jar!/:2.1.0]
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:2722)
~[ignite-core-2.1.0.jar!/:2.1.0]
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:1172)
~[ignite-core-2.1.0.jar!/:2.1.0]

our JVM arguments:

-Xms10g
-Xmx10g
-server
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=D:\data\memory
-XX:NewRatio=3
-XX:SurvivorRatio=8
-XX:+UseTLAB
-XX:GCTimeRatio=4
-XX:+ScavengeBeforeFullGC
-XX:G1HeapRegionSize=8M
-XX:ConcGCThreads=8
-XX:G1HeapWastePercent=10
-XX:+AggressiveOpts
-XX:MaxMetaspaceSize=256m
-XX:+UseG1GC
-XX:MaxGCPauseMillis=1000
-XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC
-Djava.awt.headless=true


Need we detect the Node stopping itself events?  then re-bring up the Ignite instance and
cache?  very appreciate your time!



Regards



aaron@tophold.com
Mime
View raw message