ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ght230 <ght...@163.com>
Subject Re: java.lang.IllegalStateException: Cache has been stopped
Date Wed, 09 Mar 2016 06:28:03 GMT
following is the detailed log, thank you for help me to check it.

2016-03-04 20:39:26 STDIO [ERROR] Mar 04, 2016 8:39:26 PM
org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send local partition map to node [node=TcpDiscoveryNode
[id=6701f10e-8319-47ac-99b2-3521cfaf91ca, addrs=[127.0.0.1, 192.168.70.19,
192.168.100.19, 192.168.200.19, 192.168.210.19],
sockAddrs=[HDD019/192.168.70.19:47500, /127.0.0.1:47500,
/192.168.200.19:47500, /192.168.70.19:47500, /192.168.100.19:47500,
/192.168.100.19:47500, /192.168.210.19:47500, /192.168.200.19:47500,
/192.168.210.19:47500], discPort=47500, order=75, intOrder=39,
lastExchangeTime=1457092565397, loc=false, ver=1.4.1#20150925-sha1:f051e49f,
isClient=false], exchId=null]
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=6701f10e-8319-47ac-99b2-3521cfaf91ca, addrs=[127.0.0.1, 192.168.70.19,
192.168.100.19, 192.168.200.19, 192.168.210.19],
sockAddrs=[HDD019/192.168.70.19:47500, /127.0.0.1:47500,
/192.168.200.19:47500, /192.168.70.19:47500, /192.168.100.19:47500,
/192.168.100.19:47500, /192.168.210.19:47500, /192.168.200.19:47500,
/192.168.210.19:47500], discPort=47500, order=75, intOrder=39,
lastExchangeTime=1457092565397, loc=false, ver=1.4.1#20150925-sha1:f051e49f,
isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsSingleMessage
[parts={-2100569601=GridDhtPartitionMap
[nodeId=6704f7fd-a285-401c-8563-1c704b8f0b08, updateSeq=996, moving=0,
size=100], 689859866=GridDhtPartitionMap
[nodeId=6704f7fd-a285-401c-8563-1c704b8f0b08, updateSeq=928, moving=0,
size=28], 1325947219=GridDhtPartitionMap
[nodeId=6704f7fd-a285-401c-8563-1c704b8f0b08, updateSeq=918, moving=0,
size=20]}, client=false, super=GridDhtPartitionsAbstractMessage
[exchId=null, lastVer=GridCacheVersion [topVer=68560288, nodeOrderDrId=115,
globalTime=1457095166517, order=1457168152279], super=GridCacheMessage
[msgId=369, depInfo=null, err=null, skipPrepare=false]]], policy=2]
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1071)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1214)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:873)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendLocalPartitions(GridCachePartitionExchangeManager.java:760)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:671)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:690)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1800(GridCachePartitionExchangeManager.java:95)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1152)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=6701f10e-8319-47ac-99b2-3521cfaf91ca, addrs=[127.0.0.1, 192.168.70.19,
192.168.100.19, 192.168.200.19, 192.168.210.19],
sockAddrs=[HDD019/192.168.70.19:47500, /127.0.0.1:47500,
/192.168.200.19:47500, /192.168.70.19:47500, /192.168.100.19:47500,
/192.168.100.19:47500, /192.168.210.19:47500, /192.168.200.19:47500,
/192.168.210.19:47500], discPort=47500, order=75, intOrder=39,
lastExchangeTime=1457092565397, loc=false, ver=1.4.1#20150925-sha1:f051e49f,
isClient=false]
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1940)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1880)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1066)
	... 9 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each GridComputeTask and
GridCacheTransaction has a timeout set in order to prevent parties from
waiting forever in case of network issues
[nodeId=6701f10e-8319-47ac-99b2-3521cfaf91ca,
addrs=[HDD019/192.168.70.19:47100, /192.168.200.19:47100,
/192.168.100.19:47100, /192.168.210.19:47100, /127.0.0.1:47100]]
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2421)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2074)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:1978)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$4500(TcpCommunicationSpi.java:234)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processRecovery(TcpCommunicationSpi.java:3122)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:2995)
	at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: HDD019/192.168.70.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 6 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 6 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /192.168.200.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 6 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 6 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /192.168.100.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 6 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 6 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /192.168.210.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 6 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 6 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /127.0.0.1:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 6 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID
is not as expected [expected=6701f10e-8319-47ac-99b2-3521cfaf91ca,
rcvd=6704f7fd-a285-401c-8563-1c704b8f0b08]
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2539)
		at org.apache.ignite.spi.communication.tcp.Tcp
2016-03-04 20:39:26 STDIO [ERROR]
CommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 6 more
2016-03-04 20:39:51 STDIO [ERROR] Mar 04, 2016 8:39:51 PM
org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send local partition map to node [node=TcpDiscoveryNode
[id=6701f10e-8319-47ac-99b2-3521cfaf91ca, addrs=[127.0.0.1, 192.168.70.19,
192.168.100.19, 192.168.200.19, 192.168.210.19],
sockAddrs=[HDD019/192.168.70.19:47500, /127.0.0.1:47500,
/192.168.200.19:47500, /192.168.70.19:47500, /192.168.100.19:47500,
/192.168.100.19:47500, /192.168.210.19:47500, /192.168.200.19:47500,
/192.168.210.19:47500], discPort=47500, order=75, intOrder=39,
lastExchangeTime=1457092565397, loc=false, ver=1.4.1#20150925-sha1:f051e49f,
isClient=false], exchId=null]
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=6701f10e-8319-47ac-99b2-3521cfaf91ca, addrs=[127.0.0.1, 192.168.70.19,
192.168.100.19, 192.168.200.19, 192.168.210.19],
sockAddrs=[HDD019/192.168.70.19:47500, /127.0.0.1:47500,
/192.168.200.19:47500, /192.168.70.19:47500, /192.168.100.19:47500,
/192.168.100.19:47500, /192.168.210.19:47500, /192.168.200.19:47500,
/192.168.210.19:47500], discPort=47500, order=75, intOrder=39,
lastExchangeTime=1457092565397, loc=false, ver=1.4.1#20150925-sha1:f051e49f,
isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsSingleMessage
[parts={-2100569601=GridDhtPartitionMap
[nodeId=6704f7fd-a285-401c-8563-1c704b8f0b08, updateSeq=996, moving=0,
size=100], 689859866=GridDhtPartitionMap
[nodeId=6704f7fd-a285-401c-8563-1c704b8f0b08, updateSeq=928, moving=0,
size=28], 1325947219=GridDhtPartitionMap
[nodeId=6704f7fd-a285-401c-8563-1c704b8f0b08, updateSeq=918, moving=0,
size=20]}, client=false, super=GridDhtPartitionsAbstractMessage
[exchId=null, lastVer=GridCacheVersion [topVer=68560288, nodeOrderDrId=115,
globalTime=1457095191445, order=1457168166103], super=GridCacheMessage
[msgId=370, depInfo=null, err=null, skipPrepare=false]]], policy=2]
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1071)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1214)
	at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:873)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendLocalPartitions(GridCachePartitionExchangeManager.java:760)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:671)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:690)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1800(GridCachePartitionExchangeManager.java:95)
	at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1152)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=6701f10e-8319-47ac-99b2-3521cfaf91ca, addrs=[127.0.0.1, 192.168.70.19,
192.168.100.19, 192.168.200.19, 192.168.210.19],
sockAddrs=[HDD019/192.168.70.19:47500, /127.0.0.1:47500,
/192.168.200.19:47500, /192.168.70.19:47500, /192.168.100.19:47500,
/192.168.100.19:47500, /192.168.210.19:47500, /192.168.200.19:47500,
/192.168.210.19:47500], discPort=47500, order=75, intOrder=39,
lastExchangeTime=1457092565397, loc=false, ver=1.4.1#20150925-sha1:f051e49f,
isClient=false]
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1940)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1880)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1066)
	... 9 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each GridComputeTask and
GridCacheTransaction has a timeout set in order to prevent parties from
waiting forever in case of network issues
[nodeId=6701f10e-8319-47ac-99b2-3521cfaf91ca,
addrs=[HDD019/192.168.70.19:47100, /192.168.200.19:47100,
/192.168.100.19:47100, /192.168.210.19:47100, /127.0.0.1:47100]]
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2421)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2074)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:1978)
	at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1914)
	... 11 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: HDD019/192.168.70.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 14 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 14 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /192.168.200.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 14 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 14 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /192.168.100.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 14 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 14 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /192.168.210.19:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 14 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read
remote node recovery handshake (connection closed).
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2634)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 14 more
	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
connect to address: /127.0.0.1:47100
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2426)
		... 14 more
	Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID
is not as expected [expected=6701f10e-8319-47ac-99b2-3521cfaf91ca,
rcvd=6704f7fd-a285-401c-8563-1c704b8f0b08]
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2539)
		at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2293)
		... 14 more
2016-03-04 20:39:52 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.IllegalStateException: Cache has been
stopped: PROD_BROAD_BAND_ALL_Cache
Caused by: java.lang.IllegalStateException: Cache has been stopped:
PROD_BROAD_BAND_ALL_Cache
	at
org.apache.ignite.internal.processors.cache.GridCacheGateway.checkState(GridCacheGateway.java:85)
~[ignite-core-1.4.1.jar:1.4.1]
	at
org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:174)
~[ignite-core-1.4.1.jar:1.4.1]
	at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:1826)
~[ignite-core-1.4.1.jar:1.4.1]
	at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:794)
~[ignite-core-1.4.1.jar:1.4.1]



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/java-lang-IllegalStateException-Cache-has-been-stopped-tp3389p3409.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message