ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alper Tekinalp <al...@evam.com>
Subject Re: Threads got stuck
Date Tue, 25 Oct 2016 08:20:15 GMT
Hi Yakov.

I should also mention that we load cache data from one server and wait the
data to be replicated to others. Can that cause such a situation, too?

On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yzhdanov@apache.org> wrote:

> Alper,
>
> There can be multiple reasons.
>
> Can you please reproduce the issue one more time, collect and share the
> following with us:
>
> 1. collect all the logs from all the nodes - clients and servers
> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>
> --Yakov
>
> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <alper@evam.com>:
>
>> Hi.
>>
>> There is also a few logs as :
>>
>>  Failed to register marshalled class for more than 10 times in a row (may
>> affect performance).
>>
>> Can it be releated?
>>
>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <alper@evam.com> wrote:
>>
>>> Hi all.
>>>
>>> We have 3 servers and cache configuration like:
>>>
>>>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>> name="DEFAULT">
>>>         <property name="atomicityMode" value="ATOMIC" />
>>>         <property name="cacheMode" value="PARTITIONED" />
>>>         <property name="memoryMode" value="OFFHEAP_TIERED" />
>>>         <property name="backups" value="1" />
>>>         <property name="affinity">
>>>             <bean class="org.apache.ignite.cache
>>> .affinity.fair.FairAffinityFunction">
>>>                 <constructor-arg index="0" type="int"
>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>>             </bean>
>>>         </property>
>>>         <!-- max memory is set to 0 means unlimited -->
>>>         <property name="offHeapMaxMemory" value="0" />
>>>         <property name="writeThrough" value="false" />
>>>         <property name="writeBehindEnabled" value="false" />
>>>         <property name="eagerTtl" value="false"/>
>>>     </bean>
>>>
>>> For our worker threads we check heartbeat and if a thread did not sent
>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>> recreate it.
>>>
>>> As I can see all our worker threads are stucked in cache.put() state and
>>> interrupted and recreated regularly.
>>>
>>> What can be the reason we are stucked at put? Following is stacktrace
>>> for interruption error.
>>>
>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>>         at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>> onvertToCacheException(GridCacheUtils.java:1502)
>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .cacheException(IgniteCacheProxy.java:2021)
>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1221)
>>>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>> te(ConfigManagerHelperEE.java:90)
>>>         at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>> fig(AbstractEventProcessor.java:175)
>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:130)
>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:80)
>>>         at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>> .java:199)
>>>         at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>> ler.java:46)
>>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenario(AbstractScenarioWorker.java:130)
>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>> vent(AsyncWorker.java:156)
>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>> cWorker.java:88)
>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>> to wait for asynchronous operation permit (thread got interrupted).
>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:747)
>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:745)
>>>         ... 14 more
>>> Caused by: java.lang.InterruptedException
>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .putAsync(GridCacheAdapter.java:2541)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .put(GridCacheAdapter.java:2215)
>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1214)
>>>         ... 11 more
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>


-- 
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>

Mime
View raw message