ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikolay Tikhonov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-4589) Possible starvation during rebalancing for marshaller cache
Date Thu, 02 Feb 2017 18:26:51 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850277#comment-15850277
] 

Nikolay Tikhonov commented on IGNITE-4589:
------------------------------------------

Cause of the problem in processing ordered messages. During balancing between two nodes happens
the following scenario: 
* Ordered {{GridDhtPartitionSupplyMessageV2}} message was sent for {{ignite-marshaller-sys-cache}}
cache with {{GridIoPolicy#MARSH_CACHE_POOL}};
* Ordered {{GridDhtPartitionSupplyMessageV2}} message was sent for {{ignite-sys-cache}} cache
with {{GridIoPolicy#UTILITY_CACHE_POOL}};
* On receiver node, the first message creates {{GridCommunicationMessageSet}} and start polling
messages from queue in marshaller thread pool.
* Second supply message added in the same {{GridCommunicationMessageSet}} and that thread
(from marshaller thread pool) is proccessed this message, although the message have different
io policy.

> Possible starvation during rebalancing for marshaller cache
> -----------------------------------------------------------
>
>                 Key: IGNITE-4589
>                 URL: https://issues.apache.org/jira/browse/IGNITE-4589
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache
>    Affects Versions: 1.8
>            Reporter: Nikolay Tikhonov
>            Assignee: Nikolay Tikhonov
>
> When Ignite processing supply messages (rebalancing) in marshaller threads and calling
marshaller cache from them, but responses also should be proccessed in marshaller threadpool.
It leads to starvation.
> {noformat}
> marshaller-cache-#26%test-node%" #117 prio=5 os_prio=0 tid=0x00007f89e40c6000 nid=0x1dc
waiting on condition [0x00007f8a3fffc000]
>    java.lang.Thread.State: WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for <0x00000000c206d318> (a org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedGetFuture)
>  at java.util.concurrent.locks.LockSupport.park(Unknown Source)
>  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
Source)
>  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown
Source)
>  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown
Source)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
>  at org.apache.ignite.internal.processors.cache.GridCacheAdapter.getTopologySafe(GridCacheAdapter.java:1340)
>  at org.apache.ignite.internal.MarshallerContextImpl.className(MarshallerContextImpl.java:195)
>  at org.apache.ignite.internal.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:174)
>  at org.apache.ignite.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:266)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:318)
>  at java.io.ObjectInputStream.readObject(Unknown Source)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:491)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:579)
>  at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:324)
>  at java.io.ObjectInputStream.readObject(Unknown Source)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:491)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:579)
>  at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
>  at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:324)
>  at java.io.ObjectInputStream.readObject(Unknown Source)
>  at org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:218)
>  at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>  at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1614)
>  at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1680)
>  at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450)
>  at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:298)
>  at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:100)
>  at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>  at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9722)
>  at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9751)
>  at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:111)
>  at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.unmarshal(CacheObjectBinaryProcessorImpl.java:811)
>  at org.apache.ignite.internal.processors.cache.CacheObjectImpl.finishUnmarshal(CacheObjectImpl.java:109)
>  at org.apache.ignite.internal.processors.cache.GridCacheEntryInfo.unmarshal(GridCacheEntryInfo.java:416)
>  at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionSupplyMessageV2.finishUnmarshal(GridDhtPartitionSupplyMessageV2.java:245)
>  at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1229)
>  at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:358)
>  at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$400(GridCacheIoManager.java:89)
>  at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1276)
>  at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1082)
>  at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:102)
>  at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2332)
>  at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1042)
>  at org.apache.ignite.internal.managers.communication.GridIoManager.access$1900(GridIoManager.java:102)
>  at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:1011)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>  at java.lang.Thread.run(Unknown Source)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message