ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From piali <piali.mazum...@gmail.com>
Subject `Unable to create cluster of server containers running on individual VMs
Date Wed, 30 Nov 2016 04:05:19 GMT
Hi,I am trying to create an apache ignite server cluster but unable to bring
it up.I have two VM running on two separate host machines and the apache
ignite server container (Docker) in running on those VMs . The VMs are
accessible using floating IPs (e.g., VM1-172.26.116.67 and
VM2-172.26.116.150) and containers are using host network. Also, I can inter
ping the containers.To start the servers, I am using the
$IGNITE_HOME/bin/ignite.sh. however, I changed the default configuration to
enable discovery. Please find below the co   /                                           
                                                                                         
                                                                                         
                                            
127.0.0.1:47100..47509                               
172.26.116.67:47100..47509                                                               
                                                                                         
                                                                               
/When I start the 1st Apache Ignite server container on VM1, I see warnings
related to remote node GC pauses even though I tuned off heap memory (), and
also, no remote node is running. (verified through visor)
[22:55:24,132][INFO][main][TcpCommunicationSpi] Successfully bound to TCP
port [port=47100,
locHost=0.0.0.0/0.0.0.0][22:55:24,854][INFO][main][TcpDiscoverySpi]
Successfully bound to TCP port [port=47500,
localHost=0.0.0.0/0.0.0.0][22:55:26,379][WARNING][main][TcpDiscoverySpi]
Timed out waiting for message to be read (most probably, the reason is in
long GC pauses on remote node)
[curTimeout=50][22:55:26,482][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=100][22:55:26,684][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=200][22:55:27,086][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=400][22:55:27,888][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=800][22:55:29,491][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=1600][22:55:32,696][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=3200][22:55:39,098][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=6400][22:55:51,904][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=12800][22:56:17,523][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=25600][22:56:18,033][WARNING][main][GridCacheProcessor] Eviction
policy not enabled with ONHEAP_TIERED mode for cache (entries will not be
moved to off-heap store):
default[22:56:18,120][SEVERE][grid-nio-worker-1-#38%null%][GridDirectParser]
Failed to read message [msg=null, buf=java.nio.DirectByteBuffer[pos=5
lim=420 cap=32768], reader=null, ses=GridSelectorNioSessionImpl
[selectorIdx=1, queueSize=1, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=5 lim=420
cap=32768], recovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:48819,
createTime=1480460126370, closeTime=0, bytesSent=0, bytesRcvd=420,
sndSchedTime=1480460178019, lastSndTime=1480460178114,
lastRcvTime=1480460178114, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@330f5ec2, directMode=true],
GridConnectionBytesVerifyFilter],
accepted=true]]]……………[22:56:18,127][WARNING][grid-nio-worker-1-#38%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=1, queueSize=1, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=5 lim=420
cap=32768], recovery=null, super=GridNioSessionImpl
[locAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:48819,
createTime=1480460126370, closeTime=0, bytesSent=0, bytesRcvd=420,
sndSchedTime=1480460178019, lastSndTime=1480460178114,
lastRcvTime=1480460178114, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@330f5ec2, directMode=true],
GridConnectionBytesVerifyFilter],
accepted=true]][22:56:18,127][SEVERE][grid-nio-worker-1-#38%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception.……………..After some times
this node starts with the below
logs[22:56:18,573][INFO][exchange-worker-#47%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=0], evt=NODE_JOINED,
node=6a09c54d-9cbd-42f8-bdb9-522dc438ce1c]………[22:56:18,633][INFO][main][GridDiscoveryManager]
Topology snapshot [ver=1, servers=1, clients=0, CPUs=2,
heap=1.0GB][22:56:28,053][INFO][ignite-update-notifier-timer][GridUpdateNotifier]
Your version is up to date.As soon as I start the 2nd Apache Ignite server
container on VM2, I get the below logs although I have increased the default
network timeout ()[23:18:10,313][INFO][main][TcpCommunicationSpi]
Successfully bound to TCP port [port=47100,
locHost=0.0.0.0/0.0.0.0][23:18:10,946][INFO][main][TcpDiscoverySpi]
Successfully bound to TCP port [port=47500,
localHost=0.0.0.0/0.0.0.0][23:18:12,410][WARNING][main][TcpDiscoverySpi]
Timed out waiting for message to be read (most probably, the reason is in
long GC pauses on remote node)
[curTimeout=50][23:18:12,512][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=100][23:18:12,715][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=200][23:18:13,117][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=400][23:18:13,919][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=800][23:18:15,523][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=1600][23:18:18,728][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=3200][23:18:25,136][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=6400][23:18:37,942][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=12800][23:19:03,569][WARNING][main][TcpDiscoverySpi] Timed out
waiting for message to be read (most probably, the reason is in long GC
pauses on remote node)
[curTimeout=25600]…………….[23:19:55,021][SEVERE][grid-nio-worker-1-#38%null%][GridDirectParser]
Failed to read message [msg=null, buf=java.nio.DirectByteBuffer[pos=5
lim=420 cap=32768], reader=null, ses=GridSelectorNioSessionImpl
[selectorIdx=1, queueSize=1, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=5 lim=420
cap=32768], recovery=null, super=GridNioSessionImpl
[locAddr=/172.20.29.33:47100, rmtAddr=/172.26.116.67:41952,
createTime=1480461492404, closeTime=0, bytesSent=0, bytesRcvd=420,
sndSchedTime=1480461594989, lastSndTime=1480461595001,
lastRcvTime=1480461595001, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@9df8cc3, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]class
org.apache.ignite.IgniteException: Invalid message type:
-84……………….[23:19:55,058][WARNING][grid-nio-worker-0-#37%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=1, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=5 lim=420
cap=32768], recovery=null, super=GridNioSessionImpl
[locAddr=/172.20.29.33:47100, rmtAddr=/172.26.116.67:54660,
createTime=1480461492362, closeTime=0, bytesSent=0, bytesRcvd=420,
sndSchedTime=1480461594989, lastSndTime=1480461595011,
lastRcvTime=1480461595011, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@9df8cc3, directMode=true],
GridConnectionBytesVerifyFilter],
accepted=true]][23:19:55,058][SEVERE][grid-nio-worker-0-#37%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception.………………
[23:19:55,610][INFO][exchange-worker-#47%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=0], evt=NODE_JOINED,
node=1af31843-483e-42b0-90e3-0afc7f772d61]………………[23:19:55,351][WARNING][main][GridCacheProcessor]
Eviction policy not enabled with ONHEAP_TIERED mode for cache (entries will
not be moved to off-heap store):
default………………….[23:19:55,810][INFO][main][GridDiscoveryManager] Topology
snapshot [ver=1, servers=1, clients=0, CPUs=2,
heap=1.0GB][23:20:05,051][INFO][ignite-update-notifier-timer][GridUpdateNotifier]
Your version is up to date.The servers are not joining.I am attaching both
the server log files for reference.Can you please help.Let me know if you
need any further information.Thanks & Regards,Piali Mazumder Nath
VM1-ignite-6a09c54d.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n9286/VM1-ignite-6a09c54d.log>
 
VM2-ignite-1af31843.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n9286/VM2-ignite-1af31843.log>
 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unable-to-create-cluster-of-server-containers-running-on-individual-VMs-tp9286.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Mime
View raw message