ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolai Tikhonov <ntikho...@apache.org>
Subject Re: Ignite for Spark on YARN Deployment
Date Thu, 11 Aug 2016 09:43:21 GMT
Hi,

Could you please provide logs from containers?
Also IGNITE_PATH property incorrect. The property should contains path to
apache ignite zip archive. For example:
/ignite/apache-ignite-fabric-1.7.0-bin.zip.
Also IGNITE_USERS_LIBS not needed here. The property need to use when you
want deploy to cluster your own libs.

On Thu, Aug 11, 2016 at 7:54 AM, percent620 <percent620@163.com> wrote:

> Thanks vkulichenko's quick response.
>
> Here is my detailed steps about how to deploy and integration spark with
> ignite as below.
>
> 1、Followed this guidelines about how to deploy ignite-yarn application.
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6941/ignite-yarn.png
> >
>
> it's successfully and the log is displayed ok
> ==================================
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/u01/hbase/hadoop-2.5.0-cdh5.3.0/share/hadoop/
> common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/
> appcache/application_1455892346017_5077/filecache/
> 10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 16/08/10 22:54:57 INFO impl.ContainerManagementProtocolProxy:
> yarn.client.max-cached-nodemanagers-proxies : 0
> Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Application master registered.
> Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 2,432, cpu 1.
> Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 2,432, cpu 1.
> 16/08/10 22:54:59 INFO impl.AMRMClientImpl: Received new token for :
> vmsecdomain010194062066.cm10:61362
> 16/08/10 22:54:59 INFO impl.AMRMClientImpl: Received new token for :
> vmsecdomain010194062042.cm10:42077
> Aug 10, 2016 10:54:59 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersAllocated
> INFO: Launching container: container_1455892346017_5077_02_000002.
> 16/08/10 22:54:59 INFO impl.ContainerManagementProtocolProxy: Opening
> proxy
> : vmsecdomain010194062066.cm10:61362
> Aug 10, 2016 10:54:59 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersAllocated
> INFO: Launching container: container_1455892346017_5077_02_000003.
> 16/08/10 22:54:59 INFO impl.ContainerManagementProtocolProxy: Opening
> proxy
> : vmsecdomain010194062042.cm10:42077
> Aug 10, 2016 10:55:08 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_1455892346017_5077_02_000002. State: COMPLETE.
> Aug 10, 2016 10:55:09 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_1455892346017_5077_02_000003. State: COMPLETE.
>
>
> 2、downloaded this apache-ignite-fabric-1.6.0-bin.zip and unzip this file
> to
> the /u01/XXX/apache-ignite-fabric-1.6.0-bin directory.
>
> 3、cluster16.properis content is as below
> $cat cluster16.properties
> # The number of nodes in the cluster.
> IGNITE_NODE_COUNT=2
> # The number of CPU Cores for each Apache Ignite node.
> IGNITE_RUN_CPU_PER_NODE=1
> # The number of Megabytes of RAM for each Apache Ignite node.
> IGNITE_MEMORY_PER_NODE=2048
> # The version ignite which will be run on nodes.
> IGNITE_VERSION=1.6.0
> IGNITE_WORK_DIR=/u01/XXX/apache-ignite-fabric-1.6.0-bin/
> IGNITE_XML_CONFIG=/ignite/release16/apache-ignite-fabric-1.6.0-bin/config/
> default-config.xml
> IGNITE_RELEASES_DIR=/ignite/release16/
> IGNITE_USERS_LIBS=/u01/XXX/apache-ignite-fabric-1.6.0-bin/libs/
> IGNITE_PATH=/ignite/release16/
>
>
> hdfs directory is as below
> ===========================
> ./hdfs dfs -ls /ignite/
> drwxr-xr-x   - hbase hbase          0 2016-08-10 17:07 /ignite/release16
> drwxr-xr-x   - hbase hbase          0 2016-08-02 06:25 /ignite/releases
> drwxr-xr-x   - hbase hbase          0 2016-08-10 17:24 /ignite/workdir
> -rw-r--r--   3 hbase hbase   27710331 2016-08-10 17:04 /ignite/yarn
> =============
> $./hdfs dfs -ls /ignite/release16
> drwxr-xr-x   - hbase hbase          0 2016-08-11 12:44
> /ignite/release16/apache-ignite-fabric-1.6.0-bin
>
> =============
>
> 4、Running spark code on yarn and the code is as below
> val igniteContext = new IgniteContext[String, BaseLine](sc,() => new
> IgniteConfiguration())
>
>
> the code is hanging on this and i think that this client can't connect this
> server
>
> 5、from yarn console and found the following error message
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/u01/hbase/hadoop-2.5.0-cdh5.3.0/share/hadoop/
> common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/
> appcache/application_1455892346017_5077/filecache/
> 10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 16/08/10 17:24:16 INFO impl.ContainerManagementProtocolProxy:
> yarn.client.max-cached-nodemanagers-proxies : 0
> Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Application master registered.
> Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 2,432, cpu 1.
> Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run
> INFO: Making request. Memory: 2,432, cpu 1.
> 16/08/10 17:24:17 INFO impl.AMRMClientImpl: Received new token for :
> vmsecdomain010194054064.cm10:56307
> 16/08/10 17:24:17 INFO impl.AMRMClientImpl: Received new token for :
> vmsecdomain010194062006.cm10:22640
> Aug 10, 2016 5:24:17 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersAllocated
> INFO: Launching container: container_1455892346017_5077_01_000002.
> 16/08/10 17:24:17 INFO impl.ContainerManagementProtocolProxy: Opening
> proxy
> : vmsecdomain010194054064.cm10:56307
> Aug 10, 2016 5:24:17 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersAllocated
> INFO: Launching container: container_1455892346017_5077_01_000003.
> 16/08/10 17:24:17 INFO impl.ContainerManagementProtocolProxy: Opening
> proxy
> : vmsecdomain010194062006.cm10:22640
> Aug 10, 2016 5:24:21 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_1455892346017_5077_01_000003. State: COMPLETE.
> Aug 10, 2016 5:24:22 PM org.apache.ignite.yarn.ApplicationMaster
> onContainersCompleted
> INFO: Container completed. Container id:
> container_1455892346017_5077_01_000002. State: COMPLETE.
> 16/08/10 22:47:26 WARN security.UserGroupInformation:
> PriviledgedActionException as:hbase (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:47:26 WARN ipc.Client: Exception encountered while connecting
> to
> the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:47:26 WARN security.UserGroupInformation:
> PriviledgedActionException as:hbase (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:47:26 INFO retry.RetryInvocationHandler: Exception while
> invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1455892346017_5077_000001
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)
>         at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:104)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.
> allocate(AMRMClientImpl.java:278)
>         at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$
> HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
>         at org.apache.hadoop.ipc.Client.call(Client.java:1411)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy12.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
>         ... 8 more
> 16/08/10 22:47:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing
> over to rm2
> 16/08/10 22:47:26 INFO retry.RetryInvocationHandler: Exception while
> invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2
> after 1 fail over attempts. Trying to fail over after sleeping for 22339ms.
> java.net.ConnectException: Call From
> vmsecdomain010194062063.cm10/10.194.62.63 to
> vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection
> exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.net.NetUtils.wrapWithMessage(
> NetUtils.java:783)
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1415)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy12.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.
> allocate(AMRMClientImpl.java:278)
>         at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$
> HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
> Caused by: java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(
> SocketChannelImpl.java:739)
>         at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(
> SocketIOWithTimeout.java:206)
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
>         at org.apache.hadoop.ipc.Client$Connection.setupConnection(
> Client.java:606)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(
> Client.java:700)
>         at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.
> java:367)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1382)
>         ... 12 more
> 16/08/10 22:47:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing
> over to rm1
> 16/08/10 22:47:48 WARN security.UserGroupInformation:
> PriviledgedActionException as:hbase (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:47:48 WARN ipc.Client: Exception encountered while connecting
> to
> the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:47:48 WARN security.UserGroupInformation:
> PriviledgedActionException as:hbase (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:47:48 INFO retry.RetryInvocationHandler: Exception while
> invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1
> after 2 fail over attempts. Trying to fail over immediately.
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1455892346017_5077_000001
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)
>         at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:104)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.
> allocate(AMRMClientImpl.java:278)
>         at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$
> HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
>         at org.apache.hadoop.ipc.Client.call(Client.java:1411)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy12.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
>         ... 8 more
> 16/08/10 22:47:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing
> over to rm2
> 16/08/10 22:47:48 INFO retry.RetryInvocationHandler: Exception while
> invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2
> after 3 fail over attempts. Trying to fail over after sleeping for 26407ms.
> java.net.ConnectException: Call From
> vmsecdomain010194062063.cm10/10.194.62.63 to
> vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection
> exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.net.NetUtils.wrapWithMessage(
> NetUtils.java:783)
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1415)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy12.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.
> allocate(AMRMClientImpl.java:278)
>         at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$
> HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
> Caused by: java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(
> SocketChannelImpl.java:739)
>         at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(
> SocketIOWithTimeout.java:206)
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
>         at org.apache.hadoop.ipc.Client$Connection.setupConnection(
> Client.java:606)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(
> Client.java:700)
>         at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.
> java:367)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1382)
>         ... 12 more
> 16/08/10 22:48:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing
> over to rm1
> 16/08/10 22:48:14 WARN security.UserGroupInformation:
> PriviledgedActionException as:hbase (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:48:14 WARN ipc.Client: Exception encountered while connecting
> to
> the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:48:14 WARN security.UserGroupInformation:
> PriviledgedActionException as:hbase (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
> 16/08/10 22:48:14 INFO retry.RetryInvocationHandler: Exception while
> invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1
> after 4 fail over attempts. Trying to fail over immediately.
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1455892346017_5077_000001
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)
>         at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:104)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.
> allocate(AMRMClientImpl.java:278)
>         at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$
> HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1455892346017_5077_000001
>         at org.apache.hadoop.ipc.Client.call(Client.java:1411)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy12.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
>         ... 8 more
> 16/08/10 22:48:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing
> over to rm2
> 16/08/10 22:48:14 INFO retry.RetryInvocationHandler: Exception while
> invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2
> after 5 fail over attempts. Trying to fail over after sleeping for 19315ms.
> java.net.ConnectException: Call From
> vmsecdomain010194062063.cm10/10.194.62.63 to
> vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection
> exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.net.NetUtils.wrapWithMessage(
> NetUtils.java:783)
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1415)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy12.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.allocate(Unknown Source)
>         at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.
> allocate(AMRMClientImpl.java:278)
>         at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$
> HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
> Caused by: java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(
> SocketChannelImpl.java:739)
>         at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(
> SocketIOWithTimeout.java:206)
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
>         at org.apache.hadoop.ipc.Client$Connection.setupConnection(
> Client.java:606)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(
> Client.java:700)
>         at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.
> java:367)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1382)
>         ... 12 more
> 16/08/10 22:48:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing
> over to rm1
>
>
> Can anyone help me on this issue? I'm vary urgent for this issues, thanks
> again!!!1
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6941.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Mime
View raw message