ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: TcpCommunicationSpi - Caught unhandled exception in NIO worker thread (restart the node) java.lang.OutOfMemoryError: Java heap space
Date Wed, 19 Jul 2017 16:12:49 GMT
Hi,

>From ignite-2.0, TcpCommunicationSpi has messageQueueLimit=0.
That means outbound message queue is unlimited. You should see a warning in
logs on node startup.

Try to set messageQueueLimit to smth >0.
It was 1024 by default in previous versions of ignite.

On Wed, Jul 19, 2017 at 12:56 PM, aaron@tophold.com <aaron@tophold.com>
wrote:

> Hi All,
>
>
>
> We have instance , in fact no operation and  data exist yet, but the
> memory usage keep going on until to 6G~
>
>
> From the heap dump most objects are GridNioRecoveryDescriptor take 4G; is
> this expected?   my configuration almost same as the default
> configuration xml + a cache with CacheJdbcPojoStore;
>
>
> while in fact there are only hundreds of data in DB now.  we keep get OOM
> exception as below, then the node just quite from the cluster.  thanks for
> your time!!!
>
>
>
>
>
>
>
>
>
> [ERROR] 2017-07-19 09:50:51.171 [grid-nio-worker-tcp-comm-4-#29%PortfolioEventIgnite%]
> [ig] TcpCommunicationSpi - Caught unhandled exception in NIO worker thread
> (restart the node).
>
> java.lang.OutOfMemoryError: Java heap space
>
>         at java.util.ArrayDeque.allocateElements(ArrayDeque.java:142)
> ~[?:1.8.0_131]
>
>         at java.util.ArrayDeque.<init>(ArrayDeque.java:198) ~[?:1.8.0_131]
>
>         at org.apache.ignite.internal.util.nio.GridNioRecoveryDescriptor.<
> init>(GridNioRecoveryDescriptor.java:99) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
> recoveryDescriptor(TcpCommunicationSpi.java:3534)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
> inRecoveryDescriptor(TcpCommunicationSpi.java:3493)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.access$1100(TcpCommunicationSpi.java:241)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.
> onFirstMessage(TcpCommunicationSpi.java:537)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.
> onMessage(TcpCommunicationSpi.java:626) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.
> onMessage(TcpCommunicationSpi.java:342) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioCodecFilter.
> onMessageReceived(GridNioCodecFilter.java:117)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.
> GridConnectionBytesVerifyFilter.onMessageReceived(
> GridConnectionBytesVerifyFilter.java:133) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> HeadFilter.onMessageReceived(GridNioServer.java:3062)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterChain.
> onMessageReceived(GridNioFilterChain.java:175)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> DirectNioClientWorker.processRead(GridNioServer.java:1121)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2031)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> AbstractNioClientWorker.bodyInternal(GridNioServer.java:1868)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> AbstractNioClientWorker.body(GridNioServer.java:1573)
> [ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [ignite-core-2.0.0.jar!/:2.0.0]
>
>         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
>
> [ERROR] 2017-07-19 09:50:51.171 [grid-nio-worker-tcp-comm-4-#29%PortfolioEventIgnite%]
> [ig] TcpCommunicationSpi - Runtime error caught during grid runnable
> execution: DirectNioClientWorker [super=Abstr
>
> actNioClientWorker [selector=sun.nio.ch.EPollSelectorImpl@11c850c6,
> idx=4, bytesRcvd=42, bytesSent=18, bytesRcvd0=42, bytesSent0=18,
> select=false, super=GridWorker [name=grid-nio-worker-tcp-comm-4, ign
>
> iteInstanceName=PortfolioEventIgnite, finished=false,
> hashCode=1422574651, interrupted=false, runner=grid-nio-worker-tcp-
> comm-4-#29%PortfolioEventIgnite%]]]
>
> java.lang.OutOfMemoryError: Java heap space
>
>         at java.util.ArrayDeque.allocateElements(ArrayDeque.java:142)
> ~[?:1.8.0_131]
>
>         at java.util.ArrayDeque.<init>(ArrayDeque.java:198) ~[?:1.8.0_131]
>
>         at org.apache.ignite.internal.util.nio.GridNioRecoveryDescriptor.<
> init>(GridNioRecoveryDescriptor.java:99) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
> recoveryDescriptor(TcpCommunicationSpi.java:3534)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
> inRecoveryDescriptor(TcpCommunicationSpi.java:3493)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.access$1100(TcpCommunicationSpi.java:241)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.
> onFirstMessage(TcpCommunicationSpi.java:537)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.
> onMessage(TcpCommunicationSpi.java:626) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.
> onMessage(TcpCommunicationSpi.java:342) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioCodecFilter.
> onMessageReceived(GridNioCodecFilter.java:117)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.
> GridConnectionBytesVerifyFilter.onMessageReceived(
> GridConnectionBytesVerifyFilter.java:133) ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> HeadFilter.onMessageReceived(GridNioServer.java:3062)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioFilterChain.
> onMessageReceived(GridNioFilterChain.java:175)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> DirectNioClientWorker.processRead(GridNioServer.java:1121)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2031)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> AbstractNioClientWorker.bodyInternal(GridNioServer.java:1868)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.nio.GridNioServer$
> AbstractNioClientWorker.body(GridNioServer.java:1573)
> ~[ignite-core-2.0.0.jar!/:2.0.0]
>
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [ignite-core-2.0.0.jar!/:2.0.0]
>
>         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> ------------------------------
> aaron@tophold.com
>



-- 
Best regards,
Andrey V. Mashenkov

Mime
View raw message