ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From percent620 <percent...@163.com>
Subject Re: Ignite for Spark on YARN Deployment
Date Thu, 11 Aug 2016 04:54:00 GMT
Thanks vkulichenko's quick response.

Here is my detailed steps about how to deploy and integration spark with
ignite as below.

1、Followed this guidelines about how to deploy ignite-yarn application.
<http://apache-ignite-users.70518.x6.nabble.com/file/n6941/ignite-yarn.png> 

it's successfully and the log is displayed ok
==================================
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/u01/hbase/hadoop-2.5.0-cdh5.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5077/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/08/10 22:54:57 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/08/10 22:54:59 INFO impl.AMRMClientImpl: Received new token for :
vmsecdomain010194062066.cm10:61362
16/08/10 22:54:59 INFO impl.AMRMClientImpl: Received new token for :
vmsecdomain010194062042.cm10:42077
Aug 10, 2016 10:54:59 PM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1455892346017_5077_02_000002.
16/08/10 22:54:59 INFO impl.ContainerManagementProtocolProxy: Opening proxy
: vmsecdomain010194062066.cm10:61362
Aug 10, 2016 10:54:59 PM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1455892346017_5077_02_000003.
16/08/10 22:54:59 INFO impl.ContainerManagementProtocolProxy: Opening proxy
: vmsecdomain010194062042.cm10:42077
Aug 10, 2016 10:55:08 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1455892346017_5077_02_000002. State: COMPLETE.
Aug 10, 2016 10:55:09 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1455892346017_5077_02_000003. State: COMPLETE.


2、downloaded this apache-ignite-fabric-1.6.0-bin.zip and unzip this file to
the /u01/XXX/apache-ignite-fabric-1.6.0-bin directory.

3、cluster16.properis content is as below
$cat cluster16.properties 
# The number of nodes in the cluster.
IGNITE_NODE_COUNT=2
# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1
# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=2048
# The version ignite which will be run on nodes.
IGNITE_VERSION=1.6.0
IGNITE_WORK_DIR=/u01/XXX/apache-ignite-fabric-1.6.0-bin/
IGNITE_XML_CONFIG=/ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml
IGNITE_RELEASES_DIR=/ignite/release16/
IGNITE_USERS_LIBS=/u01/XXX/apache-ignite-fabric-1.6.0-bin/libs/
IGNITE_PATH=/ignite/release16/


hdfs directory is as below
===========================
./hdfs dfs -ls /ignite/
drwxr-xr-x   - hbase hbase          0 2016-08-10 17:07 /ignite/release16
drwxr-xr-x   - hbase hbase          0 2016-08-02 06:25 /ignite/releases
drwxr-xr-x   - hbase hbase          0 2016-08-10 17:24 /ignite/workdir
-rw-r--r--   3 hbase hbase   27710331 2016-08-10 17:04 /ignite/yarn
=============
$./hdfs dfs -ls /ignite/release16
drwxr-xr-x   - hbase hbase          0 2016-08-11 12:44
/ignite/release16/apache-ignite-fabric-1.6.0-bin

=============

4、Running spark code on yarn and the code is as below
val igniteContext = new IgniteContext[String, BaseLine](sc,() => new
IgniteConfiguration())


the code is hanging on this and i think that this client can't connect this
server

5、from yarn console and found the following error message
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/u01/hbase/hadoop-2.5.0-cdh5.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/appcache/application_1455892346017_5077/filecache/10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/08/10 17:24:16 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Application master registered.
Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run
INFO: Making request. Memory: 2,432, cpu 1.
16/08/10 17:24:17 INFO impl.AMRMClientImpl: Received new token for :
vmsecdomain010194054064.cm10:56307
16/08/10 17:24:17 INFO impl.AMRMClientImpl: Received new token for :
vmsecdomain010194062006.cm10:22640
Aug 10, 2016 5:24:17 PM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1455892346017_5077_01_000002.
16/08/10 17:24:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy
: vmsecdomain010194054064.cm10:56307
Aug 10, 2016 5:24:17 PM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1455892346017_5077_01_000003.
16/08/10 17:24:17 INFO impl.ContainerManagementProtocolProxy: Opening proxy
: vmsecdomain010194062006.cm10:22640
Aug 10, 2016 5:24:21 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1455892346017_5077_01_000003. State: COMPLETE.
Aug 10, 2016 5:24:22 PM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1455892346017_5077_01_000002. State: COMPLETE.
16/08/10 22:47:26 WARN security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:47:26 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:47:26 WARN security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:47:26 INFO retry.RetryInvocationHandler: Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1.
Trying to fail over immediately.
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1455892346017_5077_000001
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy13.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
	at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy12.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
	... 8 more
16/08/10 22:47:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm2
16/08/10 22:47:26 INFO retry.RetryInvocationHandler: Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2
after 1 fail over attempts. Trying to fail over after sleeping for 22339ms.
java.net.ConnectException: Call From
vmsecdomain010194062063.cm10/10.194.62.63 to
vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see: 
http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
	at org.apache.hadoop.ipc.Client.call(Client.java:1415)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy12.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy13.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
	at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
	at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
	at org.apache.hadoop.ipc.Client.call(Client.java:1382)
	... 12 more
16/08/10 22:47:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm1
16/08/10 22:47:48 WARN security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:47:48 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:47:48 WARN security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:47:48 INFO retry.RetryInvocationHandler: Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1
after 2 fail over attempts. Trying to fail over immediately.
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1455892346017_5077_000001
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy13.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
	at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy12.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
	... 8 more
16/08/10 22:47:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm2
16/08/10 22:47:48 INFO retry.RetryInvocationHandler: Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2
after 3 fail over attempts. Trying to fail over after sleeping for 26407ms.
java.net.ConnectException: Call From
vmsecdomain010194062063.cm10/10.194.62.63 to
vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see: 
http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
	at org.apache.hadoop.ipc.Client.call(Client.java:1415)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy12.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy13.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
	at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
	at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
	at org.apache.hadoop.ipc.Client.call(Client.java:1382)
	... 12 more
16/08/10 22:48:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm1
16/08/10 22:48:14 WARN security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:48:14 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:48:14 WARN security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
16/08/10 22:48:14 INFO retry.RetryInvocationHandler: Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1
after 4 fail over attempts. Trying to fail over immediately.
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1455892346017_5077_000001
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy13.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
	at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1455892346017_5077_000001
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy12.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
	... 8 more
16/08/10 22:48:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm2
16/08/10 22:48:14 INFO retry.RetryInvocationHandler: Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2
after 5 fail over attempts. Trying to fail over after sleeping for 19315ms.
java.net.ConnectException: Call From
vmsecdomain010194062063.cm10/10.194.62.63 to
vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see: 
http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
	at org.apache.hadoop.ipc.Client.call(Client.java:1415)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy12.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy13.allocate(Unknown Source)
	at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
	at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
	at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
	at org.apache.hadoop.ipc.Client.call(Client.java:1382)
	... 12 more
16/08/10 22:48:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm1


Can anyone help me on this issue? I'm vary urgent for this issues, thanks
again!!!1



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6941.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message