ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From percent620 <percent...@163.com>
Subject Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production
Date Tue, 23 Aug 2016 09:13:21 GMT
I have fixed the file issues
"file:/usr/apache-ignite-fabric-1.6.0-bin/config/default-config.xml" on
today.

I verified again on our production env and found another issues as below
1) I have set IGNTE_HOME on spark driver machine and start to submit this
jar to yarn cluster 
export IGNITE_HOME=/u01/xxxx/apache-ignite-fabric-1.6.0-bin

$/u01/spark-1.6.0-hive/bin/spark-submit --driver-memory 4G --class
com.ValidSparkCache --master yarn --executor-cores 5 --executor-memory
15000m --num-executors 50 --conf spark.rdd.compress=false --conf
spark.shuffle.compress=false --conf spark.broadcast.compress=false
/u01/xxxx/spark_zmqpull_engine847.jar 

16/08/23 16:42:07 WARN Utils: Service 'SparkUI' could not bind on port 4040.
Attempting port 4041.
16/08/23 16:42:07 WARN Utils: Service 'SparkUI' could not bind on port 4041.
Attempting port 4042.
16/08/23 16:42:07 WARN Utils: Service 'SparkUI' could not bind on port 4042.
Attempting port 4043.
16/08/23 16:42:07 WARN Utils: Service 'SparkUI' could not bind on port 4043.
Attempting port 4044.
define igniteContext start===
define igniteContext end===
[Stage 0:=======================>                                  (4 + 6) /
10][16:43:17] New version is available at ignite.apache.org: 1.7.0
initalRDD.couner=/. 100000      partition=> 10                                  
*16/08/23 16:43:21 WARN TaskSetManager: Lost task 7.0 in stage 1.0 (TID 17,
xxx): class org.apache.ignite.IgniteCheckedException: Spring XML
configuration path is invalid: config/default-config.xml. Note that this
path should be either absolute or a relative local file system path,
relative to META-INF in classpath or valid URL to IGNITE_HOME.
*	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3580)
	at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:678)
	at
org.apache.ignite.internal.IgnitionEx.loadConfiguration(IgnitionEx.java:717)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:198)
	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:138)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:171)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:170)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:756)
Caused by: java.net.MalformedURLException: no protocol:
config/default-config.xml
	at java.net.URL.<init>(URL.java:585)
	at java.net.URL.<init>(URL.java:482)
	at java.net.URL.<init>(URL.java:431)
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3571)
	... 18 more

[Stage 1:>                                                        (0 + 10) /
10]16/08/23 16:43:31 WARN TaskSetManager: Lost task 3.2 in stage 1.0 (TID
23, xxxx): class org.apache.ignite.IgniteCheckedException: Spring XML
configuration path is invalid: config/default-config.xml. Note that this
path should be either absolute or a relative local file system path,
relative to META-INF in classpath or valid URL to IGNITE_HOME.
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3580)
	at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:678)
	at
org.apache.ignite.internal.IgnitionEx.loadConfiguration(IgnitionEx.java:717)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:198)
	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:138)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:171)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:170)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:756)
Caused by: java.net.MalformedURLException: no protocol:
config/default-config.xml
	at java.net.URL.<init>(URL.java:585)
	at java.net.URL.<init>(URL.java:482)
	at java.net.URL.<init>(URL.java:431)
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3571)
	... 18 more

16/08/23 16:43:31 ERROR TaskSetManager: Task 3 in stage 1.0 failed 4 times;
aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 3 in stage 1.0 failed 4 times, most recent failure:
Lost task 3.3 in stage 1.0 (TID 25, xxx): class
org.apache.ignite.IgniteCheckedException: Spring XML configuration path is
invalid: config/default-config.xml. Note that this path should be either
absolute or a relative local file system path, relative to META-INF in
classpath or valid URL to IGNITE_HOME.
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3580)
	at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:678)
	at
org.apache.ignite.internal.IgnitionEx.loadConfiguration(IgnitionEx.java:717)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:198)
	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:138)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:171)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:170)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:756)
Caused by: java.net.MalformedURLException: no protocol:
config/default-config.xml
	at java.net.URL.<init>(URL.java:585)
	at java.net.URL.<init>(URL.java:482)
	at java.net.URL.<init>(URL.java:431)
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3571)
	... 18 more

Driver stacktrace:
	at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at scala.Option.foreach(Option.scala:236)
	at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
	at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
	at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
	at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:920)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:918)
	at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:918)
	at org.apache.ignite.spark.IgniteRDD.savePairs(IgniteRDD.scala:170)
	at
com.alibaba.aim.sparkrule.search.es.ValidSparkCache$.main(TestSparkCache.scala:99)
	at
com.alibaba.aim.sparkrule.search.es.ValidSparkCache.main(TestSparkCache.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: class org.apache.ignite.IgniteCheckedException: Spring XML
configuration path is invalid: config/default-config.xml. Note that this
path should be either absolute or a relative local file system path,
relative to META-INF in classpath or valid URL to IGNITE_HOME.
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3580)
	at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:678)
	at
org.apache.ignite.internal.IgnitionEx.loadConfiguration(IgnitionEx.java:717)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at
org.apache.ignite.spark.IgniteContext$$anonfun$$lessinit$greater$2.apply(IgniteContext.scala:85)
	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:198)
	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:138)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:171)
	at
org.apache.ignite.spark.IgniteRDD$$anonfun$savePairs$1.apply(IgniteRDD.scala:170)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:756)
Caused by: java.net.MalformedURLException: no protocol:
config/default-config.xml
	at java.net.URL.<init>(URL.java:585)
	at java.net.URL.<init>(URL.java:482)
	at java.net.URL.<init>(URL.java:431)
	at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3571)
	... 18 more


2) this error message display this  config/default-config.xml is not existed
on other spark workers
3)attache one captain logs as below

/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/ignite/apache-ignite-fabric-1.6.0-bin/work/log/ignite-28cf1142.0.log
[16:34:44,787][INFO][main][IgniteKernal] 

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 1.6.0#20160518-sha1:0b22c45b
>>> 2016 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

[16:34:44,789][INFO][main][IgniteKernal] Config URL:
file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/./ignite-config.xml
[16:34:44,789][INFO][main][IgniteKernal] Daemon mode: off
[16:34:44,789][INFO][main][IgniteKernal] OS: Linux
2.6.32-220.23.2.ali878.el6.x86_64 amd64
[16:34:44,789][INFO][main][IgniteKernal] OS user: hbase
[16:34:44,790][INFO][main][IgniteKernal] Language runtime: Java Platform API
Specification ver. 1.7
[16:34:44,790][INFO][main][IgniteKernal] VM information: Java(TM) SE Runtime
Environment 1.7.0_79-b15 Oracle Corporation OpenJDK (Alibaba) 64-Bit Server
VM 24.79-b02-internal
[16:34:44,792][INFO][main][IgniteKernal] VM total memory: 1.9GB
[16:34:44,792][INFO][main][IgniteKernal] Remote Management [restart: on,
REST: on, JMX (remote: on, port: 49112, auth: off, ssl: off)]
[16:34:44,792][INFO][main][IgniteKernal]
IGNITE_HOME=/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/ignite/apache-ignite-fabric-1.6.0-bin
[16:34:44,793][INFO][main][IgniteKernal] VM arguments: [-Xms1g, -Xmx1g,
-XX:+AggressiveOpts, -XX:MaxPermSize=256m, -DIGNITE_QUIET=true,
-DIGNITE_SUCCESS_FILE=/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/ignite/apache-ignite-fabric-1.6.0-bin/work/ignite_success_42f8b644-7d8f-46db-931d-71a7bf7bdf59,
-Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49112,
-Dcom.sun.management.jmxremote.authenticate=false,
-Dcom.sun.management.jmxremote.ssl=false,
-DIGNITE_HOME=/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/ignite/apache-ignite-fabric-1.6.0-bin,
-DIGNITE_PROG_NAME=./ignite/apache-ignite-fabric-1.6.0-bin/bin/ignite.sh,
-Xmx2048m, -Xms2048m]
[16:34:44,793][INFO][main][IgniteKernal] Configured caches
['ignite-marshaller-sys-cache', 'ignite-sys-cache',
'ignite-atomics-sys-cache']
[16:34:44,797][INFO][main][IgniteKernal] 3-rd party licenses can be found
at:
/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/ignite/apache-ignite-fabric-1.6.0-bin/libs/licenses
[16:34:44,889][INFO][main][IgniteKernal] Non-loopback local IPs: xxx
[16:34:44,889][INFO][main][IgniteKernal] Enabled local MACs: 283152A7FE8E
[16:34:44,909][INFO][main][IgnitePluginProcessor] Configured plugins:
[16:34:44,909][INFO][main][IgnitePluginProcessor]   ^-- None
[16:34:44,909][INFO][main][IgnitePluginProcessor] 
[16:34:44,991][INFO][main][TcpCommunicationSpi] IPC shared memory server
endpoint started [port=48100,
tokDir=/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5426/container_1455892346017_5426_01_000002/ignite/apache-ignite-fabric-1.6.0-bin/work/ipc/shmem/28cf1142-e0f7-475d-91bd-694bfa96d122-119317]
[16:34:44,991][INFO][main][TcpCommunicationSpi] Successfully bound shared
memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
[16:34:45,012][INFO][main][TcpCommunicationSpi] Successfully bound to TCP
port [port=47100, locHost=0.0.0.0/0.0.0.0]
[16:34:45,036][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled
(to enable configure any GridCheckpointSpi implementation)
[16:34:45,078][WARNING][main][GridCollisionManager] Collision resolution is
disabled (all jobs will be activated upon arrival).
[16:34:45,083][WARNING][main][NoopSwapSpaceSpi] Swap space is disabled. To
enable use FileSwapSpaceSpi.
[16:34:45,086][INFO][main][IgniteKernal] Security status
[authentication=off, tls/ssl=off]
[16:34:45,448][INFO][main][GridTcpRestProtocol] Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[16:34:45,553][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
[port=47500, localHost=0.0.0.0/0.0.0.0]
[16:41:33,493][INFO][main][GridCacheProcessor] Started cache
[name=ignite-marshaller-sys-cache, mode=REPLICATED]
[16:41:33,531][INFO][main][GridCacheProcessor] Started cache
[name=ignite-atomics-sys-cache, mode=PARTITIONED]
[16:41:33,533][INFO][main][GridCacheProcessor] Started cache
[name=ignite-sys-cache, mode=REPLICATED]
[16:41:33,849][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Cache rebalancing scheduled: [cache=ignite-sys-cache, waitList=[]]
[16:41:33,850][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Cache rebalancing scheduled: [cache=ignite-marshaller-sys-cache,
waitList=[]]
[16:41:33,850][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Cache rebalancing scheduled: [cache=ignite-atomics-sys-cache,
waitList=[ignite-sys-cache, ignite-marshaller-sys-cache]]
[16:41:33,851][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Rebalancing required [top=AffinityTopologyVersion [topVer=5, minorTopVer=0],
evt=NODE_JOINED, node=28cf1142-e0f7-475d-91bd-694bfa96d122]
[16:41:33,851][INFO][exchange-worker-#114%null%][GridDhtPartitionDemander]
<ignite-marshaller-sys-cache> Starting rebalancing
[cache=ignite-marshaller-sys-cache, mode=SYNC,
fromNode=dafb84e5-37d8-48af-b7b3-1eed5f9e357d, partitionsCount=11,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=1]
[16:41:33,852][INFO][exchange-worker-#114%null%][GridDhtPartitionDemander]
<ignite-marshaller-sys-cache> Starting rebalancing
[cache=ignite-marshaller-sys-cache, mode=SYNC,
fromNode=5cec9032-9274-4bb8-98e0-d8c891761030, partitionsCount=9,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=1]
[16:41:33,860][INFO][sys-#52%null%][GridDhtPartitionDemander]
<ignite-sys-cache> Starting rebalancing [cache=ignite-sys-cache, mode=SYNC,
fromNode=dafb84e5-37d8-48af-b7b3-1eed5f9e357d, partitionsCount=49,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=1]
[16:41:33,861][INFO][sys-#52%null%][GridDhtPartitionDemander]
<ignite-sys-cache> Starting rebalancing [cache=ignite-sys-cache, mode=SYNC,
fromNode=5cec9032-9274-4bb8-98e0-d8c891761030, partitionsCount=51,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=1]
[16:41:33,862][INFO][marshaller-cache-#116%null%][GridDhtPartitionDemander]
<ignite-marshaller-sys-cache> Completed rebalancing
[cache=ignite-marshaller-sys-cache,
fromNode=dafb84e5-37d8-48af-b7b3-1eed5f9e357d,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], time=11 ms]
[16:41:33,863][INFO][marshaller-cache-#117%null%][GridDhtPartitionDemander]
<ignite-marshaller-sys-cache> Completed (final) rebalancing
[cache=ignite-marshaller-sys-cache,
fromNode=5cec9032-9274-4bb8-98e0-d8c891761030,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], time=11 ms]
[16:41:33,871][INFO][utility-#118%null%][GridDhtPartitionDemander]
<ignite-sys-cache> Completed rebalancing [cache=ignite-sys-cache,
fromNode=5cec9032-9274-4bb8-98e0-d8c891761030,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], time=21 ms]
[16:41:33,872][INFO][utility-#119%null%][GridDhtPartitionDemander]
<ignite-sys-cache> Completed (final) rebalancing [cache=ignite-sys-cache,
fromNode=dafb84e5-37d8-48af-b7b3-1eed5f9e357d,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], time=21 ms]
[16:41:33,872][INFO][sys-#52%null%][GridDhtPartitionDemander]
<ignite-atomics-sys-cache> Starting rebalancing
[cache=ignite-atomics-sys-cache, mode=SYNC,
fromNode=dafb84e5-37d8-48af-b7b3-1eed5f9e357d, partitionsCount=182,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=1]
[16:41:33,873][INFO][sys-#52%null%][GridDhtPartitionDemander]
<ignite-atomics-sys-cache> Starting rebalancing
[cache=ignite-atomics-sys-cache, mode=SYNC,
fromNode=5cec9032-9274-4bb8-98e0-d8c891761030, partitionsCount=162,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=1]
[16:41:33,907][INFO][sys-#51%null%][GridDhtPartitionDemander]
<ignite-atomics-sys-cache> Completed rebalancing
[cache=ignite-atomics-sys-cache,
fromNode=5cec9032-9274-4bb8-98e0-d8c891761030,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], time=30 ms]
[16:41:33,909][INFO][sys-#53%null%][GridDhtPartitionDemander]
<ignite-atomics-sys-cache> Completed (final) rebalancing
[cache=ignite-atomics-sys-cache,
fromNode=dafb84e5-37d8-48af-b7b3-1eed5f9e357d,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], time=30 ms]
[16:41:33,965][INFO][main][IgniteKernal] To start Console Management &
Monitoring run ignitevisorcmd.{sh|bat}
[16:41:33,966][INFO][main][IgniteKernal] 
[16:41:33,970][INFO][main][IgniteKernal] 

>>> +----------------------------------------------------------------------+
>>> Ignite ver. 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf8045ff34a031
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 2.6.32-220.23.2.ali878.el6.x86_64 amd64
>>> CPU(s): 24
>>> Heap: 2.0GB
>>> VM name: 119317@xxxx
>>> Local node [ID=28CF1142-E0F7-475D-91BD-694BFA96D122, order=5,
>>> clientMode=false]
>>> Local node addresses: [xxxx/yyy, /127.0.0.1]
>>> Local ports: TCP:11211 TCP:47100 TCP:47500 TCP:48100 

[16:41:33,972][INFO][main][GridDiscoveryManager] Topology snapshot [ver=5,
servers=3, clients=0, CPUs=72, heap=6.0GB]
[16:41:35,506][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=5, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=dafb84e5-37d8-48af-b7b3-1eed5f9e357d]
[16:41:35,754][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Added new node to topology: TcpDiscoveryNode
[id=08e62f09-26a7-4a7e-b649-7c0439974c9c, addrs=[xxx, 127.0.0.1],
sockAddrs=[xxx/xxx:47500, /xxx:47500, /127.0.0.1:47500], discPort=47500,
order=6, intOrder=5, lastExchangeTime=1471941695581, loc=false,
ver=1.6.0#20160518-sha1:0b22c45b, isClient=false]
[16:41:35,756][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Topology snapshot [ver=6, servers=4, clients=0, CPUs=96, heap=8.0GB]
[16:41:36,485][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=6, minorTopVer=0], evt=NODE_JOINED,
node=08e62f09-26a7-4a7e-b649-7c0439974c9c]
[16:41:38,683][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=6, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=dafb84e5-37d8-48af-b7b3-1eed5f9e357d]
[16:42:33,979][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:01:00:015]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.07%, avg=0.28%, GC=0%]
    ^-- Heap [used=417MB, free=78.72%, comm=1963MB]
    ^-- Non heap [used=30MB, free=90.08%, comm=30MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:42:43,210][INFO][ignite-update-notifier-timer][GridUpdateNotifier]
Update status is not available.
[16:43:07,517][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Added new node to topology: TcpDiscoveryNode
[id=f6ff2e3e-6b6f-4b2e-a7c3-07e5526edd09, addrs=[xxx, 127.0.0.1],
sockAddrs=[xxx/yyy:0, /xxx:0, /127.0.0.1:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1471941787467, loc=false,
ver=1.6.0#20160518-sha1:0b22c45b, isClient=true]
[16:43:07,520][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=4, clients=1, CPUs=120, heap=12.0GB]
[16:43:07,537][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=7, minorTopVer=0], evt=NODE_JOINED,
node=f6ff2e3e-6b6f-4b2e-a7c3-07e5526edd09]
[16:43:33,987][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:02:00:020]
    ^-- H/N/C [hosts=5, nodes=5, CPUs=120]
    ^-- CPU [cur=0.07%, avg=0.17%, GC=0%]
    ^-- Heap [used=427MB, free=78.25%, comm=1963MB]
    ^-- Non heap [used=30MB, free=90.03%, comm=31MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:43:35,292][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Node left topology: TcpDiscoveryNode
[id=f6ff2e3e-6b6f-4b2e-a7c3-07e5526edd09, addrs=[xxx, 127.0.0.1],
sockAddrs=[xxx/xxx:0, /xxxx:0, /127.0.0.1:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1471941787467, loc=false,
ver=1.6.0#20160518-sha1:0b22c45b, isClient=true]
[16:43:35,294][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Topology snapshot [ver=8, servers=4, clients=0, CPUs=96, heap=8.0GB]
[16:43:35,312][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=8, minorTopVer=0], evt=NODE_LEFT,
node=f6ff2e3e-6b6f-4b2e-a7c3-07e5526edd09]
[16:44:33,994][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:03:00:028]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.14%, GC=0%]
    ^-- Heap [used=441MB, free=77.52%, comm=1963MB]
    ^-- Non heap [used=30MB, free=89.94%, comm=31MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:45:34,005][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:04:00:036]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.12%, GC=0%]
    ^-- Heap [used=448MB, free=77.16%, comm=1963MB]
    ^-- Non heap [used=30MB, free=89.93%, comm=31MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:46:34,012][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:05:00:048]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=455MB, free=76.82%, comm=1963MB]
    ^-- Non heap [used=30MB, free=89.92%, comm=31MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:46:50,900][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Added new node to topology: TcpDiscoveryNode
[id=488d4aff-09b1-463d-aaa6-656f8adda0a4, addrs=[xxx, 127.0.0.1],
sockAddrs=[xxx/xxx:0, /xxx:0, /127.0.0.1:0], discPort=0, order=9,
intOrder=7, lastExchangeTime=1471942010840, loc=false,
ver=1.6.0#20160518-sha1:0b22c45b, isClient=true]
[16:46:50,902][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Topology snapshot [ver=9, servers=4, clients=1, CPUs=120, heap=9.0GB]
[16:46:50,915][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=9, minorTopVer=0], evt=NODE_JOINED,
node=488d4aff-09b1-463d-aaa6-656f8adda0a4]
[16:47:13,185][INFO][exchange-worker-#114%null%][GridCacheProcessor] Started
cache [name=partitioned, mode=PARTITIONED]
[16:47:13,476][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=9, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=488d4aff-09b1-463d-aaa6-656f8adda0a4]
[16:47:34,015][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:06:00:050]
    ^-- H/N/C [hosts=5, nodes=5, CPUs=120]
    ^-- CPU [cur=0.07%, avg=0.1%, GC=0%]
    ^-- Heap [used=472MB, free=75.93%, comm=1963MB]
    ^-- Non heap [used=30MB, free=89.82%, comm=31MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:48:15,510][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Node left topology: TcpDiscoveryNode
[id=488d4aff-09b1-463d-aaa6-656f8adda0a4, addrs=[xxx, 127.0.0.1],
sockAddrs=[xxx/xxx:0, /xx:0, /127.0.0.1:0], discPort=0, order=9, intOrder=7,
lastExchangeTime=1471942010840, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=true]
[16:48:15,515][INFO][disco-event-worker-#112%null%][GridDiscoveryManager]
Topology snapshot [ver=10, servers=4, clients=0, CPUs=96, heap=8.0GB]
[16:48:15,543][INFO][exchange-worker-#114%null%][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=10, minorTopVer=0], evt=NODE_LEFT,
node=488d4aff-09b1-463d-aaa6-656f8adda0a4]
[16:48:34,022][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:07:00:058]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.14%, GC=0%]
    ^-- Heap [used=201MB, free=89.74%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.46%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:49:34,031][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:08:00:065]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.13%, GC=0%]
    ^-- Heap [used=208MB, free=89.39%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.46%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:50:34,037][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:09:00:072]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.12%, GC=0%]
    ^-- Heap [used=215MB, free=89.04%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.45%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:51:34,046][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:10:00:078]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.11%, GC=0%]
    ^-- Heap [used=221MB, free=88.71%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.44%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:52:34,052][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:11:00:081]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=228MB, free=88.34%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.41%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:53:34,056][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:12:00:089]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=235MB, free=88.01%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.4%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:54:34,062][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:13:00:093]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
    ^-- Heap [used=242MB, free=87.66%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.4%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:55:34,067][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:14:00:102]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.09%, GC=0%]
    ^-- Heap [used=248MB, free=87.33%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.4%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:56:34,074][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:15:00:106]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.08%, GC=0%]
    ^-- Heap [used=256MB, free=86.96%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.4%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:57:34,079][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:16:00:114]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.08%, GC=0%]
    ^-- Heap [used=262MB, free=86.63%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.39%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:57:52,170][WARNING][tcp-disco-sock-reader-#11%null%][TcpDiscoverySpi]
Unknown connection detected (is some other software connecting to this
Ignite port? missing SSL configuration on remote node?) [rmtAddr=/127.0.0.1]
[16:58:34,084][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:17:00:118]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.08%, GC=0%]
    ^-- Heap [used=270MB, free=86.24%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.38%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[16:59:34,090][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:18:00:125]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.08%, GC=0%]
    ^-- Heap [used=276MB, free=85.91%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.37%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:00:34,096][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:19:00:128]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.07%, GC=0%]
    ^-- Heap [used=283MB, free=85.55%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.37%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:01:34,103][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:20:00:134]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0%, avg=0.07%, GC=0%]
    ^-- Heap [used=290MB, free=85.2%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.37%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:02:34,110][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:21:00:141]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.07%, avg=0.07%, GC=0%]
    ^-- Heap [used=297MB, free=84.86%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.36%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:03:34,113][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:22:00:151]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.07%, GC=0%]
    ^-- Heap [used=305MB, free=84.46%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.36%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:04:34,123][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:23:00:156]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.07%, GC=0%]
    ^-- Heap [used=311MB, free=84.12%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.35%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:05:34,128][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:24:00:162]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0%, avg=0.06%, GC=0%]
    ^-- Heap [used=318MB, free=83.76%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.35%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:06:34,136][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:25:00:167]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.06%, GC=0%]
    ^-- Heap [used=325MB, free=83.43%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.35%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:07:34,137][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:26:00:168]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.07%, avg=0.06%, GC=0%]
    ^-- Heap [used=332MB, free=83.08%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.34%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[17:08:34,136][INFO][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=28cf1142, name=null, uptime=00:27:00:170]
    ^-- H/N/C [hosts=4, nodes=4, CPUs=96]
    ^-- CPU [cur=0.03%, avg=0.06%, GC=0%]
    ^-- Heap [used=338MB, free=82.74%, comm=1963MB]
    ^-- Non heap [used=32MB, free=89.33%, comm=32MB]
    ^-- Public thread pool [active=0, idle=48, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]


4)Does I need to copy config/default.xml to all the spark workers? I think
NO.





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205p7236.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Mime
View raw message