ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alper Tekinalp <al...@evam.com>
Subject Re: Threads got stuck
Date Wed, 26 Oct 2016 06:52:48 GMT
Hi.

After restarting servers, that issue did not occur again. I guess for some
reason the cluster did not start normally.

Thanks responding so quick.

Regards.

On Tue, Oct 25, 2016 at 11:31 AM, Yakov Zhdanov <yzhdanov@apache.org> wrote:

> Alper, thanks for clarification, this will definitely help after we get the
> info I requested. This is the only way to go with the investigation.
>
> --Yakov
>
> 2016-10-25 11:20 GMT+03:00 Alper Tekinalp <alper@evam.com>:
>
> > Hi Yakov.
> >
> > I should also mention that we load cache data from one server and wait
> the
> > data to be replicated to others. Can that cause such a situation, too?
> >
> > On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yzhdanov@apache.org>
> > wrote:
> >
> >> Alper,
> >>
> >> There can be multiple reasons.
> >>
> >> Can you please reproduce the issue one more time, collect and share the
> >> following with us:
> >>
> >> 1. collect all the logs from all the nodes - clients and servers
> >> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
> >>
> >> --Yakov
> >>
> >> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <alper@evam.com>:
> >>
> >>> Hi.
> >>>
> >>> There is also a few logs as :
> >>>
> >>>  Failed to register marshalled class for more than 10 times in a row
> >>> (may affect performance).
> >>>
> >>> Can it be releated?
> >>>
> >>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <alper@evam.com>
> wrote:
> >>>
> >>>> Hi all.
> >>>>
> >>>> We have 3 servers and cache configuration like:
> >>>>
> >>>>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
> >>>> name="DEFAULT">
> >>>>         <property name="atomicityMode" value="ATOMIC" />
> >>>>         <property name="cacheMode" value="PARTITIONED" />
> >>>>         <property name="memoryMode" value="OFFHEAP_TIERED" />
> >>>>         <property name="backups" value="1" />
> >>>>         <property name="affinity">
> >>>>             <bean class="org.apache.ignite.cache
> >>>> .affinity.fair.FairAffinityFunction">
> >>>>                 <constructor-arg index="0" type="int"
> >>>> value="#{evamProperties['topology.cache.partition.size']}"/>
> >>>>             </bean>
> >>>>         </property>
> >>>>         <!-- max memory is set to 0 means unlimited -->
> >>>>         <property name="offHeapMaxMemory" value="0" />
> >>>>         <property name="writeThrough" value="false" />
> >>>>         <property name="writeBehindEnabled" value="false" />
> >>>>         <property name="eagerTtl" value="false"/>
> >>>>     </bean>
> >>>>
> >>>> For our worker threads we check heartbeat and if a thread did not sent
> >>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
> >>>> recreate it.
> >>>>
> >>>> As I can see all our worker threads are stucked in cache.put() state
> >>>> and interrupted and recreated regularly.
> >>>>
> >>>> What can be the reason we are stucked at put? Following is stacktrace
> >>>> for interruption error.
> >>>>
> >>>> javax.cache.CacheException: class org.apache.ignite.
> IgniteInterruptedException:
> >>>> Failed to wait for asynchronous operation permit (thread got
> interrupted).
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheUtils.c
> >>>> onvertToCacheException(GridCacheUtils.java:1502)
> >>>>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .cacheException(IgniteCacheProxy.java:2021)
> >>>>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1221)
> >>>>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.
> setSta
> >>>> te(ConfigManagerHelperEE.java:90)
> >>>>         at com.intellica.project.helper.ee.
> StateMachineConfigManagerEEI
> >>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
> >>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeCon
> >>>> fig(AbstractEventProcessor.java:175)
> >>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:130)
> >>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:80)
> >>>>         at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine
> >>>> .java:199)
> >>>>         at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHand
> >>>> ler.java:46)
> >>>>         at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
> >>>>         at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenario(AbstractScenarioWorker.java:130)
> >>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.
> processE
> >>>> vent(AsyncWorker.java:156)
> >>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> Asyn
> >>>> cWorker.java:88)
> >>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
> >>>> to wait for asynchronous operation permit (thread got interrupted).
> >>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:747)
> >>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:745)
> >>>>         ... 14 more
> >>>> Caused by: java.lang.InterruptedException
> >>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquir
> >>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
> >>>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .putAsync(GridCacheAdapter.java:2541)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .put(GridCacheAdapter.java:2215)
> >>>>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1214)
> >>>>         ... 11 more
> >>>>
> >>>>
> >>>> --
> >>>> Alper Tekinalp
> >>>>
> >>>> Software Developer
> >>>> Evam Streaming Analytics
> >>>>
> >>>> Atatürk Mah. Turgut Özal Bulv.
> >>>> Gardenya 5 Plaza K:6 Ataşehir
> >>>> 34758 İSTANBUL
> >>>>
> >>>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
> >>>> www.evam.com.tr
> >>>> <http://www.evam.com>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alper Tekinalp
> >>>
> >>> Software Developer
> >>> Evam Streaming Analytics
> >>>
> >>> Atatürk Mah. Turgut Özal Bulv.
> >>> Gardenya 5 Plaza K:6 Ataşehir
> >>> 34758 İSTANBUL
> >>>
> >>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
> >>> www.evam.com.tr
> >>> <http://www.evam.com>
> >>>
> >>
> >>
> >
> >
> > --
> > Alper Tekinalp
> >
> > Software Developer
> > Evam Streaming Analytics
> >
> > Atatürk Mah. Turgut Özal Bulv.
> > Gardenya 5 Plaza K:6 Ataşehir
> > 34758 İSTANBUL
> >
> > Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
> > www.evam.com.tr
> > <http://www.evam.com>
> >
>



-- 
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message