Return-Path: X-Original-To: apmail-ignite-user-archive@minotaur.apache.org Delivered-To: apmail-ignite-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 215B018682 for ; Tue, 19 Jan 2016 09:26:45 +0000 (UTC) Received: (qmail 26671 invoked by uid 500); 19 Jan 2016 09:26:45 -0000 Delivered-To: apmail-ignite-user-archive@ignite.apache.org Received: (qmail 26628 invoked by uid 500); 19 Jan 2016 09:26:45 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 26618 invoked by uid 99); 19 Jan 2016 09:26:45 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2016 09:26:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8A94CC0044 for ; Tue, 19 Jan 2016 09:26:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.92 X-Spam-Level: X-Spam-Status: No, score=0.92 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id H9OEJLz7cp82 for ; Tue, 19 Jan 2016 09:26:30 +0000 (UTC) Received: from mbob.nabble.com (mbob.nabble.com [162.253.133.15]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTP id C6FD6215D4 for ; Tue, 19 Jan 2016 09:26:29 +0000 (UTC) Received: from malf.nabble.com (unknown [162.253.133.59]) by mbob.nabble.com (Postfix) with ESMTP id A560D1E90F40 for ; Tue, 19 Jan 2016 01:22:58 -0800 (PST) Date: Tue, 19 Jan 2016 01:15:31 -0800 (PST) From: Ambha To: user@ignite.apache.org Message-ID: <1453194931262-2634.post@n6.nabble.com> In-Reply-To: <1453112651642-2609.post@n6.nabble.com> References: <1453095043407-2600.post@n6.nabble.com> <1453112651642-2609.post@n6.nabble.com> Subject: Re: issue with 2-node ignite server cluster MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit This is the error it displays ********** class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.186.247], sockAddrs=[/192.168.186.247:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.186.247:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1453209329890, loc=false, ver=1.4.0#20150924-sha1:c2def5f6, isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsSingleMessage [parts={-2100569601=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=122, moving=0, size=100], 689859866=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=534, moving=0, size=512], 1325947219=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=42, moving=0, size=20], 2034899268=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=534, moving=0, size=512], -1559869495=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=534, moving=0, size=512]}, client=false, super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=GridCacheVersion [topVer=0, nodeOrderDrId=0, globalTime=0, order=1453209329443], super=GridCacheMessage [msgId=26, depInfo=null, err=null, skipPrepare=false]]], policy=2] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1071) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1214) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:873) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendLocalPartitions(GridCachePartitionExchangeManager.java:760) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:671) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:690) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1800(GridCachePartitionExchangeManager.java:95) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1152) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.186.247], sockAddrs=[/192.168.186.247:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.186.247:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1453209329890, loc=false, ver=1.4.0#20150924-sha1:c2def5f6, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1940) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1880) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1066) ... 9 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each GridComputeTask and GridCacheTransaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[/192.168.186.247:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2421) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2074) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:1978) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1914) ... 11 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /192.168.186.247:47100 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read remote node recovery handshake (connection closed). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293) ... 14 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /0:0:0:0:0:0:0:1%lo:47100 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is not as expected [expected=4b13c224-21be-4c32-b29d-a0d7d079df10, rcvd=3955b599-e49c-43f3-ad07-752b349db1ac] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2539) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293) ... 14 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /127.0.0.1:47100 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is not as expected [expected=4b13c224-21be-4c32-b29d-a0d7d079df10, rcvd=3955b599-e49c-43f3-ad07-752b349db1ac] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2539) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293) ... 14 more [18:47:25,778][SEVERE][exchange-worker-#41%testGrid%][GridCachePartitionExchangeManager] Failed to send local partition map to node [node=TcpDiscoveryNode [id=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.186.247], sockAddrs=[/192.168.186.247:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.186.247:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1453209329890, loc=false, ver=1.4.0#20150924-sha1:c2def5f6, isClient=false], exchId=null] class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.186.247], sockAddrs=[/192.168.186.247:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.186.247:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1453209329890, loc=false, ver=1.4.0#20150924-sha1:c2def5f6, isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsSingleMessage [parts={-2100569601=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=122, moving=0, size=100], 689859866=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=534, moving=0, size=512], 1325947219=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=42, moving=0, size=20], 2034899268=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=534, moving=0, size=512], -1559869495=GridDhtPartitionMap [nodeId=3955b599-e49c-43f3-ad07-752b349db1ac, updateSeq=534, moving=0, size=512]}, client=false, super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=GridCacheVersion [topVer=0, nodeOrderDrId=0, globalTime=0, order=1453209329443], super=GridCacheMessage [msgId=27, depInfo=null, err=null, skipPrepare=false]]], policy=2] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1071) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1214) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:873) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendLocalPartitions(GridCachePartitionExchangeManager.java:760) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:671) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:690) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1800(GridCachePartitionExchangeManager.java:95) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1152) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.186.247], sockAddrs=[/192.168.186.247:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.186.247:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1453209329890, loc=false, ver=1.4.0#20150924-sha1:c2def5f6, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1940) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1880) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1066) ... 9 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each GridComputeTask and GridCacheTransaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=4b13c224-21be-4c32-b29d-a0d7d079df10, addrs=[/192.168.186.247:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2421) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2074) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:1978) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1914) ... 11 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /192.168.186.247:47100 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read remote node recovery handshake (connection closed). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293) ... 14 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /0:0:0:0:0:0:0:1%lo:47100 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is not as expected [expected=4b13c224-21be-4c32-b29d-a0d7d079df10, rcvd=3955b599-e49c-43f3-ad07-752b349db1ac] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2539) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293) ... 14 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /127.0.0.1:47100 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426) ... 14 more Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is not as expected [expected=4b13c224-21be-4c32-b29d-a0d7d079df10, rcvd=3955b599-e49c-43f3-ad07-752b349db1ac] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2539) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293) ... 14 more [18:47:29] Topology snapshot [ver=3, servers=2, clients=1, CPUs=5, heap=9.3GB] [18:47:29] Topology snapshot [ver=4, servers=1, clients=1, CPUs=3, heap=5.3GB] [18:47:29] Topology snapshot [ver=7, servers=1, clients=0, CPUs=1, heap=4.0GB] [18:47:31] Ignite node stopped OK [uptime=00:01:55:788] -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/issue-with-2-node-ignite-server-cluster-tp2600p2634.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.