ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alper Tekinalp <al...@evam.com>
Subject Re: Threads got stuck
Date Tue, 25 Oct 2016 07:49:21 GMT
Hi.

There is also a few logs as :

 Failed to register marshalled class for more than 10 times in a row (may
affect performance).

Can it be releated?

On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <alper@evam.com> wrote:

> Hi all.
>
> We have 3 servers and cache configuration like:
>
>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
> name="DEFAULT">
>         <property name="atomicityMode" value="ATOMIC" />
>         <property name="cacheMode" value="PARTITIONED" />
>         <property name="memoryMode" value="OFFHEAP_TIERED" />
>         <property name="backups" value="1" />
>         <property name="affinity">
>             <bean class="org.apache.ignite.cache.affinity.fair.
> FairAffinityFunction">
>                 <constructor-arg index="0" type="int"
> value="#{evamProperties['topology.cache.partition.size']}"/>
>             </bean>
>         </property>
>         <!-- max memory is set to 0 means unlimited -->
>         <property name="offHeapMaxMemory" value="0" />
>         <property name="writeThrough" value="false" />
>         <property name="writeBehindEnabled" value="false" />
>         <property name="eagerTtl" value="false"/>
>     </bean>
>
> For our worker threads we check heartbeat and if a thread did not sent
> heart beat for 10 minutes we consider it as stucked and interrrupt and
> recreate it.
>
> As I can see all our worker threads are stucked in cache.put() state and
> interrupted and recreated regularly.
>
> What can be the reason we are stucked at put? Following is stacktrace for
> interruption error.
>
> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
> Failed to wait for asynchronous operation permit (thread got interrupted).
>         at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1502)
>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2021)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(
> ConfigManagerHelperEE.java:90)
>         at com.intellica.project.helper.ee.StateMachineConfigManagerEEImp
> l.store(StateMachineConfigManagerEEImpl.java:53)
>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeConfig(AbstractEventProcessor.java:175)
>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:130)
>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:80)
>         at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine.java:199)
>         at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHandler.java:46)
>         at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenarioLogic(AbstractScenarioWorker.java:172)
>         at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenario(AbstractScenarioWorker.java:130)
>         at com.intellica.evam.engine.server.worker.AsyncWorker.
> processEvent(AsyncWorker.java:156)
>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> AsyncWorker.java:88)
> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
> wait for asynchronous operation permit (thread got interrupted).
>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:747)
>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:745)
>         ... 14 more
> Caused by: java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter.
> asyncOpAcquire(GridCacheAdapter.java:4597)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.put(GridCacheAdapter.java:2215)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
>         ... 11 more
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>



-- 
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message