ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vijayendra bhati <veejayend...@yahoo.com>
Subject Cache Connection taking time while using IgniteRDD on Spark executor
Date Wed, 27 Apr 2016 12:31:28 GMT
Hi,
I am using IgniteRDD to query data from Ignite In memory cache cluster.I have setup the cluster
on AWS. My Spark and Ignite Nodes are running on same AWS nodes.
>From the logs I can see that it take around 3-5 seconds on every executor to establish
connection with Ignite Cluster.Is there is any way to make it faster.
Please find the below logs indicating task started at 10:32:41 and till 10:32:45 it was trying
to make connection with Ignite cluster.Is there is anyway to improve this ?



16/04/27 10:32:41 INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from URL
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/./riskAnalytics-0.0.1-SNAPSHOT.jar!/config/aws-cloud/risk-analytics-cache-client-aws.xml]
16/04/27 10:32:41 INFO support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@6e7a6652:
startup date [Wed Apr 27 10:32:41 UTC 2016]; root of context hierarchy
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: 

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 1.6.0-SNAPSHOT#19700101-sha1:DEV
>>> 2016 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Config URL: n/a
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Daemon mode: off
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: OS: Linux 4.1.17-22.30.amzn1.x86_64
amd64
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: OS user: yarn
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Language runtime: Scala ver.
2.10.5
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: VM information: OpenJDK Runtime
Environment 1.8.0_77-b03 Oracle Corporation OpenJDK 64-Bit Server VM 25.77-b03
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: VM total memory: 4.9GB
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Remote Management [restart:
off, REST: on, JMX (remote: off)]
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: IGNITE_HOME=null
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: VM arguments: [-XX:OnOutOfMemoryError=kill
%p, -Xms5120m, -Xmx5120m, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+UseConcMarkSweepGC,
-XX:CMSInitiatingOccupancyFraction=70, -XX:MaxHeapFreeRatio=70, -XX:+CMSClassUnloadingEnabled,
-XX:OnOutOfMemoryError=kill -9 %p, -Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/tmp,
-Dspark.history.ui.port=18080, -Dspark.driver.port=50884, -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1461736583050_0013/container_1461736583050_0013_01_000002]
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Configured caches ['ignite-marshaller-sys-cache',
'ignite-sys-cache', 'ignite-atomics-sys-cache', 'default']
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Non-loopback local IPs: 172.31.11.240,
fe80:0:0:0:83f:42ff:fecd:2909%eth0
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Enabled local MACs: 0A3F42CD2909
16/04/27 10:32:42 INFO plugin.IgnitePluginProcessor: Configured plugins:
16/04/27 10:32:42 INFO plugin.IgnitePluginProcessor:   ^-- None
16/04/27 10:32:42 INFO plugin.IgnitePluginProcessor: 
16/04/27 10:32:42 INFO tcp.TcpCommunicationSpi: IPC shared memory server endpoint started
[port=48102, tokDir=/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/tmp/ignite/work/ipc/shmem/98a94f2f-4f19-4430-bfe1-930312846191-87676]
16/04/27 10:32:42 INFO tcp.TcpCommunicationSpi: Successfully bound shared memory communication
to TCP port [port=48102, locHost=0.0.0.0/0.0.0.0]
16/04/27 10:32:42 INFO tcp.TcpCommunicationSpi: Successfully bound to TCP port [port=47102,
locHost=0.0.0.0/0.0.0.0]
16/04/27 10:32:42 WARN noop.NoopCheckpointSpi: Checkpoints are disabled (to enable configure
any GridCheckpointSpi implementation)
16/04/27 10:32:42 WARN collision.GridCollisionManager: Collision resolution is disabled (all
jobs will be activated upon arrival).
16/04/27 10:32:42 WARN noop.NoopSwapSpaceSpi: Swap space is disabled. To enable use FileSwapSpaceSpi.
16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Security status [authentication=off,
tls/ssl=off]
16/04/27 10:32:43 INFO tcp.GridTcpRestProtocol: Command protocol successfully started [name=TCP
binary, host=0.0.0.0/0.0.0.0, port=11212]
16/04/27 10:32:43 WARN s3.TcpDiscoveryS3IpFinder: Amazon client configuration is not set (will
use default).
16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Added new node to topology: TcpDiscoveryNode
[id=623ea12b-cabb-4c1d-a13c-ac340c2f05f4, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.31.11.239],
sockAddrs=[ip-172-31-11-239.us-west-2.compute.internal/172.31.11.239:0, /0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.31.11.239:0], discPort=0, order=72, intOrder=40, lastExchangeTime=1461753163906,
loc=false, ver=1.6.0#19700101-sha1:00000000, isClient=true]
16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=72, servers=3,
clients=5, CPUs=56, heap=60.0GB]
16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Added new node to topology: TcpDiscoveryNode
[id=71337db2-7548-49e5-91e5-12544df08cc0, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.31.11.239],
sockAddrs=[ip-172-31-11-239.us-west-2.compute.internal/172.31.11.239:0, /0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.31.11.239:0], discPort=0, order=73, intOrder=41, lastExchangeTime=1461753163906,
loc=false, ver=1.6.0#19700101-sha1:00000000, isClient=true]
16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=73, servers=3,
clients=6, CPUs=56, heap=65.0GB]
16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Added new node to topology: TcpDiscoveryNode
[id=1894483f-1b3f-42d1-a336-9659ec81c339, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.31.11.241],
sockAddrs=[ip-172-31-11-241.us-west-2.compute.internal/172.31.11.241:0, /0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.31.11.241:0], discPort=0, order=74, intOrder=42, lastExchangeTime=1461753163906,
loc=false, ver=1.6.0#19700101-sha1:00000000, isClient=true]
16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=74, servers=3,
clients=7, CPUs=56, heap=70.0GB]
16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=ignite-sys-cache, mode=REPLICATED]
16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=ignite-atomics-sys-cache,
mode=PARTITIONED]
16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=default, mode=PARTITIONED]
16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=ignite-marshaller-sys-cache,
mode=REPLICATED]
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: Performance suggestions for
grid 'riskAnalyticsGrid' (fix if possible)
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid:   ^-- Decrease number of backups
(set 'keyBackups' to 0)
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: 
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: To start Console Management
& Monitoring run ignitevisorcmd.{sh|bat}
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: 
16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: 

>>> +------------------------------------------+
>>> Ignite ver. 1.6.0-SNAPSHOT#19700101-sha1:DEV
>>> +------------------------------------------+
>>> OS name: Linux 4.1.17-22.30.amzn1.x86_64 amd64
>>> CPU(s): 16
>>> Heap: 5.0GB
>>> VM name: 87676@ip-172-31-11-240
>>> Grid name: riskAnalyticsGrid
>>> Local node [ID=98A94F2F-4F19-4430-BFE1-930312846191, order=71, clientMode=true]
>>> Local node addresses: [ip-172-31-11-240.us-west-2.compute.internal/0:0:0:0:0:0:0:1%lo,
/127.0.0.1, /172.31.11.240]
>>> Local ports: TCP:11212 TCP:47102 TCP:48102 

16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=74, servers=3,
clients=7, CPUs=56, heap=70.0GB]
16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=StockSimulationsCacheWriterPartitioned,
mode=PARTITIONED]
16/04/27 10:32:45 INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from URL
[jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/./riskAnalytics-0.0.1-SNAPSHOT.jar!/config/aws-cloud/risk-analytics-cache-client-aws.xml]
16/04/27 10:32:45 INFO support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@7577a3f7:
startup date [Wed Apr 27 10:32:45 UTC 2016]; root of context hierarchy
16/04/27 10:32:45 INFO executor.Executor: Finished task 5.0 in stage 0.0 (TID 5). 1224 bytes
result sent to driver




Regards,Vij
Mime
View raw message