ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: Possible starvation in striped pool
Date Tue, 18 Jul 2017 12:21:40 GMT
Hi Alper,

Seems, sometime stripe-pool threads stucks for awhile due to short network
issues.
Possibly, it happens when nodes doesn't left the grid gracefully.

Try to set java.net.preferIPv4Stack=true on all nodes (incl. clients),
it can help in some cases when client is droped from grid due to IPv6
connectivity issues.

On Mon, Jul 17, 2017 at 11:04 AM, Alper Tekinalp <alper@evam.com> wrote:

> Hi Andrey.
>
> Logs for two server nodes are attached.
>
> On Fri, Jul 14, 2017 at 8:41 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi Alper,
>>
>> Stripe pool threads do message processing.
>> In stacktrace, I see node can send message to other node due to network
>> issue or node failure.
>> Thread try to reconnect to the node during failure detection timeout.
>>
>> What  Ignite version do you use?
>> Would you please share full logs?
>>
>> On Fri, Jul 14, 2017 at 1:24 PM, Alper Tekinalp <alper@evam.com> wrote:
>>
>>> Hi.
>>>
>>> What does following log means:
>>>
>>> [WARN ] 2017-07-12 23:00:50.786 [grid-timeout-worker-#71%cache-server%]
>>> G - >>> Possible starvation in striped pool: sys-stripe-10-#11%cache-server
>>> %
>>> [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE,
>>> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
>>> msg=GridNearSingleGetRequest
>>> ....
>>> 43f6-946a-6593f8dcde7d, taskNameHash=0, createTtl=0, accessTtl=-1]]]]
>>> deadlock: false
>>> completed: 699284
>>> Thread [name="sys-stripe-10-#11%cache-server%", id=44, state=WAITING,
>>> blockCnt=18197, waitCnt=642544]
>>>     Lock [object=o.a.i.spi.communication.tcp.TcpCommunicationSpi$Conn
>>> ectFuture@604f22f, ownerName=null, ownerId=-1]
>>>         at sun.misc.Unsafe.park(Native Method)
>>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java
>>> :186)
>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>>         at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter
>>> .java:161)
>>>         at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.
>>> java:119)
>>>         at o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClien
>>> t(TcpCommunicationSpi.java:2515)
>>>         at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0
>>> (TcpCommunicationSpi.java:2340)
>>>         at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(
>>> TcpCommunicationSpi.java:2304)
>>>         at o.a.i.i.managers.communication.GridIoManager.send(GridIoMana
>>> ger.java:1281)
>>>         at o.a.i.i.managers.communication.GridIoManager.sendOrderedMess
>>> age(GridIoManager.java:1517)
>>>         at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi
>>> thRetries(GridContinuousProcessor.java:1335)
>>>         at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi
>>> thRetries(GridContinuousProcessor.java:1306)
>>>         at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi
>>> thRetries(GridContinuousProcessor.java:1288)
>>>         at o.a.i.i.processors.continuous.GridContinuousProcessor.sendNo
>>> tification(GridContinuousProcessor.java:949)
>>>         at o.a.i.i.processors.continuous.GridContinuousProcessor.addNot
>>> ification(GridContinuousProcessor.java:892)
>>>         at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:814)
>>>         at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler.access$800(CacheContinuousQueryHandler.java:91)
>>>         at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler$1$1.apply(CacheContinuousQueryHandler.java:426)
>>>         at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler$1$1.apply(CacheContinuousQueryHandler.java:421)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFuture.java:450)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFuture.java:56)
>>>         at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapt
>>> er.java:321)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.registerResponse(GridDhtAtomicAbstract
>>> UpdateFuture.java:343)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.onResult(GridDhtAtomicAbstractUpdateFu
>>> ture.java:399)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cCache.processDhtAtomicDeferredUpdateResponse(GridDhtAtomicC
>>> ache.java:3388)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cCache.access$2000(GridDhtAtomicCache.java:126)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cCache$10.apply(GridDhtAtomicCache.java:412)
>>>         at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cCache$10.apply(GridDhtAtomicCache.java:407)
>>>         at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(G
>>> ridCacheIoManager.java:827)
>>>         at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridC
>>> acheIoManager.java:369)
>>>         at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(Gr
>>> idCacheIoManager.java:293)
>>>         at o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridC
>>> acheIoManager.java:95)
>>>         at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(Grid
>>> CacheIoManager.java:238)
>>>         at o.a.i.i.managers.communication.GridIoManager.invokeListener(
>>> GridIoManager.java:1215)
>>>         at o.a.i.i.managers.communication.GridIoManager.processRegularM
>>> essage0(GridIoManager.java:843)
>>>         at o.a.i.i.managers.communication.GridIoManager.access$2100(Gri
>>> dIoManager.java:108)
>>>         at o.a.i.i.managers.communication.GridIoManager$6.run(GridIoMan
>>> ager.java:783)
>>>         at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java
>>> :428)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> We encounter that log time to time specially when loading big
>>> partitioned caches across the cluster. What does stripes threads do? Is
>>> there a way to avoid that?
>>>
>>> Regards.
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>



-- 
Best regards,
Andrey V. Mashenkov

Mime
View raw message