ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladislav Pyatkov (Jira)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-7913) Current implementation of Internal Diagnostics may cause OOM on server nodes.
Date Tue, 22 Oct 2019 09:31:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956858#comment-16956858
] 

Vladislav Pyatkov commented on IGNITE-7913:
-------------------------------------------

In particular diagnostic procedure can lead starvation in system pool.
In my case all pool's threads are blocked on execution diagnostic:
{noforamt}
"sys-#452677%DPL_GRID%DplGridNodeName%" #974743 prio=5 os_prio=0 tid=0x00007f79b070d000 nid=0xa1333
waiting on condition [0x00007f79615f5000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00007f8003cd3238> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4299)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.toString(GridDhtCacheEntry.java:816)
	at java.lang.String.valueOf(String.java:2994)
	at org.apache.ignite.internal.util.tostring.CircularStringBuilder.append(CircularStringBuilder.java:84)
	at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:94)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:841)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1046)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:984)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:746)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:714)
	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxEntry.toString(IgniteTxEntry.java:1273)
	at java.lang.String.valueOf(String.java:2994)
	at org.apache.ignite.internal.util.tostring.CircularStringBuilder.append(CircularStringBuilder.java:84)
	at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:94)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:841)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:799)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.addCollection(GridToStringBuilder.java:892)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:837)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:799)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.appendVals(GridToStringBuilder.java:1641)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1049)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:984)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:746)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:714)
	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:466)
	at java.lang.String.valueOf(String.java:2994)
	at org.apache.ignite.internal.util.tostring.CircularStringBuilder.append(CircularStringBuilder.java:84)
	at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:94)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:841)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1046)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:984)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:695)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:654)
	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:1488)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:941)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:657)
	at java.lang.String.valueOf(String.java:2994)
	at org.apache.ignite.internal.util.tostring.CircularStringBuilder.append(CircularStringBuilder.java:84)
	at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:94)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:841)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1046)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:984)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:746)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:714)
	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter$TxFinishFuture.toString(IgniteTxAdapter.java:2444)
	at java.lang.String.valueOf(String.java:2994)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at java.util.AbstractCollection.toString(AbstractCollection.java:462)
	at java.lang.String.valueOf(String.java:2994)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at org.apache.ignite.internal.processors.cache.CacheObjectsReleaseFuture.toString(CacheObjectsReleaseFuture.java:58)
	at java.lang.String.valueOf(String.java:2994)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at java.util.AbstractCollection.toString(AbstractCollection.java:462)
	at java.lang.String.valueOf(String.java:2994)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at org.apache.ignite.internal.processors.cache.CacheObjectsReleaseFuture.toString(CacheObjectsReleaseFuture.java:58)
	at java.lang.String.valueOf(String.java:2994)
	at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
	at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:841)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1046)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:984)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:552)
	at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:491)
	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.toString(GridDhtPartitionsExchangeFuture.java:4408)
	at java.lang.String.valueOf(String.java:2994)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at org.apache.ignite.internal.IgniteDiagnosticMessage.dumpExchangeInfo(IgniteDiagnosticMessage.java:459)
	at org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.apply(IgniteDiagnosticPrepareContext.java:194)
	at org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.apply(IgniteDiagnosticPrepareContext.java:163)
	at org.apache.ignite.internal.processors.cluster.ClusterProcessor$2.onMessage(ClusterProcessor.java:198)
	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{noformat}

> Current implementation of Internal Diagnostics may cause OOM on server nodes.
> -----------------------------------------------------------------------------
>
>                 Key: IGNITE-7913
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7913
>             Project: Ignite
>          Issue Type: Improvement
>    Affects Versions: 2.3
>            Reporter: Alexei Scherbakov
>            Priority: Major
>             Fix For: 2.9
>
>
> If many transactions are active in grid, Internal Diagnostics can cause OOM on server
nodes serving IgniteDiagnosticMessage because of heap buffering.
> See the stack trace demonstrating the issue:
> {noformat}
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1012)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:762)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:710)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.toString(GridDhtCacheEntry.java:818)
>         at java.lang.String.valueOf(String.java:2994)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:826)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:783)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxEntry.toString(IgniteTxEntry.java:1267)
>         at java.lang.String.valueOf(String.java:2994)
>         at java.lang.StringBuilder.append(StringBuilder.java:131)
>         at java.util.AbstractMap.toString(AbstractMap.java:559)
>         at java.lang.String.valueOf(String.java:2994)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:864)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxRemoteStateImpl.toString(IgniteTxRemoteStateImpl.java:180)
>         at java.lang.String.valueOf(String.java:2994)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:826)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:783)
>         at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.toString(GridDistributedTxRemoteAdapter.java:926)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxRemote.toString(GridDhtTxRemote.java:373)
>         at java.lang.String.valueOf(String.java:2994)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:826)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:783)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter$TxFinishFuture.toString(IgniteTxAdapter.java:2405)
>         at java.lang.String.valueOf(String.java:2994)
>         at java.lang.StringBuilder.append(StringBuilder.java:131)
>         at java.util.AbstractCollection.toString(AbstractCollection.java:462)
>         at java.lang.String.valueOf(String.java:2994)
>         at java.lang.StringBuilder.append(StringBuilder.java:131)
>         at org.apache.ignite.internal.processors.cache.CacheObjectsReleaseFuture.toString(CacheObjectsReleaseFuture.java:58)
>         at java.lang.String.valueOf(String.java:2994)
>         at java.lang.StringBuilder.append(StringBuilder.java:131)
>         at java.util.AbstractCollection.toString(AbstractCollection.java:462)
>         at java.lang.String.valueOf(String.java:2994)
>         at java.lang.StringBuilder.append(StringBuilder.java:131)
>         at org.apache.ignite.internal.processors.cache.CacheObjectsReleaseFuture.toString(CacheObjectsReleaseFuture.java:58)
>         at java.lang.String.valueOf(String.java:2994)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:685)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:621)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.toString(GridDhtPartitionsExchangeFuture.java:3519)
>         at java.lang.String.valueOf(String.java:2994)
>         at java.lang.StringBuilder.append(StringBuilder.java:131)
>         at org.apache.ignite.internal.IgniteDiagnosticMessage$ExchangeInfoClosure.apply(IgniteDiagnosticMessage.java:368)
>         at org.apache.ignite.internal.IgniteDiagnosticMessage$ExchangeInfoClosure.apply(IgniteDiagnosticMessage.java:346)
>         at org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.moreInfo(IgniteDiagnosticPrepareContext.java:220)
>         at org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.apply(IgniteDiagnosticPrepareContext.java:202)
>         at org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.apply(IgniteDiagnosticPrepareContext.java:163)
>         at org.apache.ignite.internal.processors.cluster.ClusterProcessor$2.onMessage(ClusterProcessor.java:175)
>         at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
>         at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
>         at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
>         at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We must reconsider way to build string representation of transactions and use streaming-like
approach.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message