ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: Re: TcpCommunicationSpi - Caught unhandled exception in NIO worker thread (restart the node) java.lang.OutOfMemoryError: Java heap space
Date Thu, 20 Jul 2017 14:36:13 GMT
Hi,

Is it possible nodes can't see each other via communication ports or use
different IP protocol (IPv4 and IPv6)?
Can you share grid configuration and logs?

On Thu, Jul 20, 2017 at 7:19 AM, aaron@tophold.com <aaron@tophold.com>
wrote:

> Thanks Andrew, But really Ignite have so many communication message
> between nodes.
> I know default the includeEventTypes is empty, but if I open the debug
> model, still saw a lot information changes between nodes. there are so many
> even I can see a lot Timeout even we just keep a empty grid running...
>
> [DEBUG] 2017-07-20 02:05:16.894 [grid-timeout-worker-#23%WebIgnite%] [ig]
> GridTimeoutProcessor - Timeout has occurred: GridCommunicationMessageSet [
> nodeId=eedf67fb-6a01-4eb3-b4ac-e23446181ca9, endTime=
> 1500516316887, timeoutId=24a304d5d51-49983eab-7747-
> 47a8-912d-d06b45404385, topic=T4 [topic=TOPIC_CACHE, id1=
> 5dc785b4-dd3d-3c3b-b270-c5fe2d7ed9a2, id2=e90fb911-
> c52d-440a-a651-8f624ac7ece5, id3=1], plc=2, msgs=[],
> reserved=false, timeout=5000, skipOnTimeout=true, lastTs=1500508078116]
>
>
> Regards
> Aaron
> ------------------------------
> aaron@tophold.com
>
>
> *From:* Andrey Mashenkov <andrey.mashenkov@gmail.com>
> *Date:* 2017-07-20 00:12
> *To:* user <user@ignite.apache.org>
> *Subject:* Re: TcpCommunicationSpi - Caught unhandled exception in NIO
> worker thread (restart the node) java.lang.OutOfMemoryError: Java heap space
> Hi,
>
> From ignite-2.0, TcpCommunicationSpi has messageQueueLimit=0.
> That means outbound message queue is unlimited. You should see a warning
> in logs on node startup.
>
> Try to set messageQueueLimit to smth >0.
> It was 1024 by default in previous versions of ignite.
>
> On Wed, Jul 19, 2017 at 12:56 PM, aaron@tophold.com <aaron@tophold.com>
> wrote:
>
>> Hi All,
>>
>>
>>
>> We have instance , in fact no operation and  data exist yet, but the
>> memory usage keep going on until to 6G~
>>
>>
>> From the heap dump most objects are GridNioRecoveryDescriptor take 4G; is
>> this expected?   my configuration almost same as the default
>> configuration xml + a cache with CacheJdbcPojoStore;
>>
>>
>> while in fact there are only hundreds of data in DB now.  we keep get
>> OOM exception as below, then the node just quite from the cluster.  thanks
>> for your time!!!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [ERROR] 2017-07-19 09:50:51.171 [grid-nio-worker-tcp-comm-4-#29%PortfolioEventIgnite%]
>> [ig] TcpCommunicationSpi - Caught unhandled exception in NIO worker thread
>> (restart the node).
>>
>> java.lang.OutOfMemoryError: Java heap space
>>
>>         at java.util.ArrayDeque.allocateElements(ArrayDeque.java:142)
>> ~[?:1.8.0_131]
>>
>>         at java.util.ArrayDeque.<init>(ArrayDeque.java:198)
>> ~[?:1.8.0_131]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioRecoveryDescripto
>> r.<init>(GridNioRecoveryDescriptor.java:99)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
>> recoveryDescriptor(TcpCommunicationSpi.java:3534)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
>> inRecoveryDescriptor(TcpCommunicationSpi.java:3493)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
>> access$1100(TcpCommunicationSpi.java:241) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$
>> 2.onFirstMessage(TcpCommunicationSpi.java:537)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$
>> 2.onMessage(TcpCommunicationSpi.java:626) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$
>> 2.onMessage(TcpCommunicationSpi.java:342) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailF
>> ilter.onMessageReceived(GridNioFilterChain.java:279)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.pro
>> ceedMessageReceived(GridNioFilterAdapter.java:109)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMes
>> sageReceived(GridNioCodecFilter.java:117) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.pro
>> ceedMessageReceived(GridNioFilterAdapter.java:109)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridConnectionBytesVerif
>> yFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:133)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.pro
>> ceedMessageReceived(GridNioFilterAdapter.java:109)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter
>> .onMessageReceived(GridNioServer.java:3062)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMes
>> sageReceived(GridNioFilterChain.java:175) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioC
>> lientWorker.processRead(GridNioServer.java:1121)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNi
>> oClientWorker.processSelectedKeysOptimized(GridNioServer.java:2031)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNi
>> oClientWorker.bodyInternal(GridNioServer.java:1868)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNi
>> oClientWorker.body(GridNioServer.java:1573)
>> [ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>> [ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
>>
>> [ERROR] 2017-07-19 09:50:51.171 [grid-nio-worker-tcp-comm-4-#29%PortfolioEventIgnite%]
>> [ig] TcpCommunicationSpi - Runtime error caught during grid runnable
>> execution: DirectNioClientWorker [super=Abstr
>>
>> actNioClientWorker [selector=sun.nio.ch.EPollSelectorImpl@11c850c6,
>> idx=4, bytesRcvd=42, bytesSent=18, bytesRcvd0=42, bytesSent0=18,
>> select=false, super=GridWorker [name=grid-nio-worker-tcp-comm-4, ign
>>
>> iteInstanceName=PortfolioEventIgnite, finished=false,
>> hashCode=1422574651, interrupted=false, runner=grid-nio-worker-tcp-com
>> m-4-#29%PortfolioEventIgnite%]]]
>>
>> java.lang.OutOfMemoryError: Java heap space
>>
>>         at java.util.ArrayDeque.allocateElements(ArrayDeque.java:142)
>> ~[?:1.8.0_131]
>>
>>         at java.util.ArrayDeque.<init>(ArrayDeque.java:198)
>> ~[?:1.8.0_131]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioRecoveryDescripto
>> r.<init>(GridNioRecoveryDescriptor.java:99)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
>> recoveryDescriptor(TcpCommunicationSpi.java:3534)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
>> inRecoveryDescriptor(TcpCommunicationSpi.java:3493)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.
>> access$1100(TcpCommunicationSpi.java:241) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$
>> 2.onFirstMessage(TcpCommunicationSpi.java:537)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$
>> 2.onMessage(TcpCommunicationSpi.java:626) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$
>> 2.onMessage(TcpCommunicationSpi.java:342) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailF
>> ilter.onMessageReceived(GridNioFilterChain.java:279)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.pro
>> ceedMessageReceived(GridNioFilterAdapter.java:109)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMes
>> sageReceived(GridNioCodecFilter.java:117) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.pro
>> ceedMessageReceived(GridNioFilterAdapter.java:109)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridConnectionBytesVerif
>> yFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:133)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.pro
>> ceedMessageReceived(GridNioFilterAdapter.java:109)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter
>> .onMessageReceived(GridNioServer.java:3062)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMes
>> sageReceived(GridNioFilterChain.java:175) ~[ignite-core-2.0.0.jar!/:2.0.
>> 0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioC
>> lientWorker.processRead(GridNioServer.java:1121)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNi
>> oClientWorker.processSelectedKeysOptimized(GridNioServer.java:2031)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNi
>> oClientWorker.bodyInternal(GridNioServer.java:1868)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNi
>> oClientWorker.body(GridNioServer.java:1573)
>> ~[ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>> [ignite-core-2.0.0.jar!/:2.0.0]
>>
>>         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
>> ------------------------------
>> aaron@tophold.com
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
>


-- 
Best regards,
Andrey V. Mashenkov

Mime
View raw message