From yarn-issues-return-170040-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org  Tue Jul 23 12:13:05 2019
Return-Path: <yarn-issues-return-170040-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [207.244.88.153])
	by mx-eu-01.ponee.io (Postfix) with SMTP id C68E11802C7
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 23 Jul 2019 14:13:04 +0200 (CEST)
Received: (qmail 19443 invoked by uid 500); 23 Jul 2019 12:13:02 -0000
Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:yarn-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:yarn-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:yarn-issues@hadoop.apache.org>
List-Id: <yarn-issues.hadoop.apache.org>
Delivered-To: mailing list yarn-issues@hadoop.apache.org
Received: (qmail 19320 invoked by uid 99); 23 Jul 2019 12:13:02 -0000
Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jul 2019 12:13:02 +0000
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
	by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8E4CFE2E86
	for <yarn-issues@hadoop.apache.org>; Tue, 23 Jul 2019 12:13:00 +0000 (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
	by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 35233265CA
	for <yarn-issues@hadoop.apache.org>; Tue, 23 Jul 2019 12:13:00 +0000 (UTC)
Date: Tue, 23 Jul 2019 12:13:00 +0000 (UTC)
From: "Babble Shack (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13246342.1563807762000.31892.1563883980216@Atlassian.JIRA>
In-Reply-To: <JIRA.13246342.1563807762000@Atlassian.JIRA>
References: <JIRA.13246342.1563807762000@Atlassian.JIRA> <JIRA.13246342.1563807762405@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (YARN-9690) Invalid AMRM token when
 distributed scheduling is enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/YARN-9690?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16890=
883#comment-16890883 ]=20

Babble Shack edited comment on YARN-9690 at 7/23/19 12:12 PM:
--------------------------------------------------------------

[~bibinchundatt]

I have tried with
{code:java}
=C2=A0 <property>
=C2=A0=C2=A0 =C2=A0=C2=A0 <name>yarn.resourcemanager.scheduler.address</nam=
e>
=C2=A0=C2=A0 =C2=A0=C2=A0 <value>0.0.0.0:8049</value>
=C2=A0 </property>{code}
As you can see in my config.

I also tried with your suggestion (localhost instead of 0.0.0.0).

but that did not work either. I have attached an additional log


The stack trace is different with the localhost modification, however the c=
ause is still "Invalid AMRMToken from appattempt_1563878096961_0001_000001"

[^distributed_log]


was (Author: babbleshack):
[~bibinchundatt]
=20
 I have tried with
{code:java}
=C2=A0 <property>
=C2=A0=C2=A0 =C2=A0=C2=A0 <name>yarn.resourcemanager.scheduler.address</nam=
e>
=C2=A0=C2=A0 =C2=A0=C2=A0 <value>0.0.0.0:8049</value>
=C2=A0 </property>{code}
As you can see in my config.

I also tried with your suggestion (localhost instead of 0.0.0.0).

but that did not work either. I have attached an additional log

[^distributed_log]

> Invalid AMRM token when distributed scheduling is enabled.
> ----------------------------------------------------------
>
>                 Key: YARN-9690
>                 URL: https://issues.apache.org/jira/browse/YARN-9690
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: distributed-scheduling, yarn
>    Affects Versions: 2.9.2, 3.1.2
>         Environment: OS: Ubuntu 18.04
> JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
>            Reporter: Babble Shack
>            Priority: Major
>         Attachments: applicationlog, distributed_log, yarn-site.xml
>
>
> Applications fail to start due to invalild AMRM from application attempt.=
=20
> I have tested this with 0/100% opportunistic maps and the same issue occu=
rs regardless.=20
> {code:java}
> <!-- yarn-site.xml -->
> -->
> <configuration>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>mapreduceyarn.nodemanager.aux-services</name>
> =C2=A0=C2=A0=C2=A0 <value>mapreduce_shuffle</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <name>yarn.resourcemanager.address</name>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <value>yarn-master-0.yarn-service.yarn:8032</va=
lue>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <name>yarn.resourcemanager.scheduler.address</n=
ame>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <value>0.0.0.0:8049</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.resourcemanager.opportunistic-container-all=
ocation.enabled</name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.nodemanager.opportunistic-containers-max-qu=
eue-length</name>
> =C2=A0=C2=A0=C2=A0 <value>10</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.nodemanager.distributed-scheduling.enabled<=
/name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.webapp.ui2.enable</name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <name>yarn.resourcemanager.resource-tracker.add=
ress</name>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <value>yarn-master-0.yarn-service.yarn:8031</va=
lue>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.log-aggregation-enable</name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <name>yarn.nodemanager.aux-services</name>
> =C2=A0=C2=A0 =C2=A0=C2=A0 <value>mapreduce_shuffle</value>
> =C2=A0 </property>
> =C2=A0 <!--disable virtual memory-->
> =C2=A0 <!--
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.nodemanager.vmem-pmem-ratio</name>
> =C2=A0=C2=A0=C2=A0 <value>1.1</value>
> =C2=A0 </property>
> =C2=A0 -->
> =C2=A0 <!-- Worker(memory)=3D8GB, Node(mem)=3D6GB, Container(mem)=3D[1,2]=
GB-->
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.nodemanager.resource.memory-mb</name>
> =C2=A0=C2=A0=C2=A0 <value>7168</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.scheduler.minimum-allocation-mb</name>
> =C2=A0=C2=A0=C2=A0 <value>3584</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.scheduler.maximum-allocation-mb</name>
> =C2=A0=C2=A0=C2=A0 <value>7168</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.app.mapreduce.am.resource.mb</name>
> =C2=A0=C2=A0=C2=A0 <value>7168</value>
> =C2=A0 </property>
> =C2=A0 <!--AM 80% of mapred container-->
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.app.mapreduce.am.command-opts</name>
> =C2=A0=C2=A0=C2=A0 <value>-Xmx5734m</value>
> =C2=A0 </property>
> =C2=A0 <!--Timeline server -->
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.timeline-service.enabled</name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.resourcemanager.system-metrics-publisher.en=
abled</name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.timeline-service.generic-application-histor=
y.enabled</name>
> =C2=A0=C2=A0=C2=A0 <value>true</value>
> =C2=A0 </property>
> =C2=A0 <property>
> =C2=A0=C2=A0=C2=A0 <name>yarn.timeline-service.bind-host</name>
> =C2=A0=C2=A0=C2=A0 <value>0.0.0.0</value>
> =C2=A0 </property>
> </configuration>
> {code}
> Relevant logs:
> {code:java}
> 2019-07-22 14:56:37,104 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm=
.RMContainerAllocator: 100% of the mappers will be scheduled using OPPORTUN=
ISTIC containers
> 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy=
: Connecting to ResourceManager at yarn-master-0.yarn-service.yarn/10.244.1=
.134:8030
> 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Excepti=
on encountered while connecting to the server : org.apache.hadoop.ipc.Remot=
eException(org.apache.hadoop.security.token.SecretManager$InvalidToken): In=
valid AMRMToken from appattempt_1563805140414_0002_000002
> 2019-07-22 14:56:37,152 ERROR [main] org.apache.hadoop.mapreduce.v2.app.r=
m.RMCommunicator: Exception while registering
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRM=
Token from appattempt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce0(Native Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce(NativeConstructorAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingConstructorAccessorImpl.newIn=
stance(DelegatingConstructorAccessorImpl.java:45)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Constructor.newInstance(Construct=
or.java:423)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateExcep=
tion(RPCUtil.java:53)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOExc=
eption(RPCUtil.java:80)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowEx=
ception(RPCUtil.java:119)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:109)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native=
 Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeM=
ethodAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del=
egatingMethodAccessorImpl.java:43)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Method.invoke(Method.java:498)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvokeMethod(RetryInvocationHandler.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeMethod(RetryInvocationHandler.java:165)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invoke(RetryInvocationHandler.java:157)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeOnce(RetryInvocationHandler.java:95)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvoke(RetryInvocationHandler.java:359)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy82.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:160)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.serviceStart(RMCommunicator.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAl=
locator.serviceStart(RMContainerAllocator.java:274)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$Cont=
ainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.CompositeService.serviceS=
tart(CompositeService.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serv=
iceStart(MRAppMaster.java:1272)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.ru=
n(MRAppMaster.java:1723)
> =C2=A0=C2=A0 =C2=A0at java.security.AccessController.doPrivileged(Native =
Method)
> =C2=A0=C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.security.UserGroupInformation.doA=
s(UserGroupInformation.java:1893)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init=
AndStartAppMaster(MRAppMaster.java:1719)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main=
(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.securi=
ty.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_156=
3805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.getRpcResponse(Client.=
java:1507)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:227)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:116)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy81.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:106)
> =C2=A0=C2=A0 =C2=A0... 25 more
> 2019-07-22 14:56:37,153 INFO [main] org.apache.hadoop.service.AbstractSer=
vice: Service RMCommunicator failed in state STARTED; cause: org.apache.had=
oop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.=
SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1563805140414=
_0002_000002
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop=
.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattem=
pt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:177)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.serviceStart(RMCommunicator.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAl=
locator.serviceStart(RMContainerAllocator.java:274)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$Cont=
ainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.CompositeService.serviceS=
tart(CompositeService.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serv=
iceStart(MRAppMaster.java:1272)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.ru=
n(MRAppMaster.java:1723)
> =C2=A0=C2=A0 =C2=A0at java.security.AccessController.doPrivileged(Native =
Method)
> =C2=A0=C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.security.UserGroupInformation.doA=
s(UserGroupInformation.java:1893)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init=
AndStartAppMaster(MRAppMaster.java:1719)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main=
(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: I=
nvalid AMRMToken from appattempt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce0(Native Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce(NativeConstructorAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingConstructorAccessorImpl.newIn=
stance(DelegatingConstructorAccessorImpl.java:45)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Constructor.newInstance(Construct=
or.java:423)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateExcep=
tion(RPCUtil.java:53)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOExc=
eption(RPCUtil.java:80)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowEx=
ception(RPCUtil.java:119)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:109)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native=
 Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeM=
ethodAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del=
egatingMethodAccessorImpl.java:43)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Method.invoke(Method.java:498)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvokeMethod(RetryInvocationHandler.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeMethod(RetryInvocationHandler.java:165)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invoke(RetryInvocationHandler.java:157)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeOnce(RetryInvocationHandler.java:95)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvoke(RetryInvocationHandler.java:359)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy82.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:160)
> =C2=A0=C2=A0 =C2=A0... 14 more
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.securi=
ty.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_156=
3805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.getRpcResponse(Client.=
java:1507)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:227)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:116)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy81.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:106)
> =C2=A0=C2=A0 =C2=A0... 25 more
> 2019-07-22 14:56:37,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm=
.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 Scheduled=
Reds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAl=
loc:0 ContRel:0 HostLocal:0 RackLocal:0
> 2019-07-22 14:56:37,154 INFO [main] org.apache.hadoop.service.AbstractSer=
vice: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAlloc=
atorRouter failed in state STARTED; cause: org.apache.hadoop.yarn.exception=
s.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$Inva=
lidToken: Invalid AMRMToken from appattempt_1563805140414_0002_000002
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop=
.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattem=
pt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:177)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.serviceStart(RMCommunicator.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAl=
locator.serviceStart(RMContainerAllocator.java:274)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$Cont=
ainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.CompositeService.serviceS=
tart(CompositeService.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serv=
iceStart(MRAppMaster.java:1272)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.ru=
n(MRAppMaster.java:1723)
> =C2=A0=C2=A0 =C2=A0at java.security.AccessController.doPrivileged(Native =
Method)
> =C2=A0=C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.security.UserGroupInformation.doA=
s(UserGroupInformation.java:1893)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init=
AndStartAppMaster(MRAppMaster.java:1719)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main=
(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: I=
nvalid AMRMToken from appattempt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce0(Native Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce(NativeConstructorAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingConstructorAccessorImpl.newIn=
stance(DelegatingConstructorAccessorImpl.java:45)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Constructor.newInstance(Construct=
or.java:423)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateExcep=
tion(RPCUtil.java:53)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOExc=
eption(RPCUtil.java:80)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowEx=
ception(RPCUtil.java:119)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:109)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native=
 Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeM=
ethodAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del=
egatingMethodAccessorImpl.java:43)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Method.invoke(Method.java:498)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvokeMethod(RetryInvocationHandler.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeMethod(RetryInvocationHandler.java:165)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invoke(RetryInvocationHandler.java:157)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeOnce(RetryInvocationHandler.java:95)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvoke(RetryInvocationHandler.java:359)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy82.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:160)
> =C2=A0=C2=A0 =C2=A0... 14 more
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.securi=
ty.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_156=
3805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.getRpcResponse(Client.=
java:1507)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:227)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:116)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy81.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:106)
> =C2=A0=C2=A0 =C2=A0... 25 more
> 2019-07-22 14:56:37,154 INFO [main] org.apache.hadoop.service.AbstractSer=
vice: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in stat=
e STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: o=
rg.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMTok=
en from appattempt_1563805140414_0002_000002
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop=
.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattem=
pt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:177)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.serviceStart(RMCommunicator.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAl=
locator.serviceStart(RMContainerAllocator.java:274)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$Cont=
ainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.CompositeService.serviceS=
tart(CompositeService.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serv=
iceStart(MRAppMaster.java:1272)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.ru=
n(MRAppMaster.java:1723)
> =C2=A0=C2=A0 =C2=A0at java.security.AccessController.doPrivileged(Native =
Method)
> =C2=A0=C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.security.UserGroupInformation.doA=
s(UserGroupInformation.java:1893)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init=
AndStartAppMaster(MRAppMaster.java:1719)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main=
(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: I=
nvalid AMRMToken from appattempt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce0(Native Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce(NativeConstructorAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingConstructorAccessorImpl.newIn=
stance(DelegatingConstructorAccessorImpl.java:45)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Constructor.newInstance(Construct=
or.java:423)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateExcep=
tion(RPCUtil.java:53)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOExc=
eption(RPCUtil.java:80)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowEx=
ception(RPCUtil.java:119)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:109)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native=
 Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeM=
ethodAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del=
egatingMethodAccessorImpl.java:43)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Method.invoke(Method.java:498)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvokeMethod(RetryInvocationHandler.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeMethod(RetryInvocationHandler.java:165)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invoke(RetryInvocationHandler.java:157)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeOnce(RetryInvocationHandler.java:95)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvoke(RetryInvocationHandler.java:359)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy82.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:160)
> =C2=A0=C2=A0 =C2=A0... 14 more
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.securi=
ty.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_156=
3805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.getRpcResponse(Client.=
java:1507)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:227)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:116)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy81.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:106)
> =C2=A0=C2=A0 =C2=A0... 25 more
> 2019-07-22 14:56:37,155 INFO [main] org.apache.hadoop.mapreduce.jobhistor=
y.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outs=
tanding queue size is 3
> 2019-07-22 14:56:37,155 INFO [main] org.apache.hadoop.mapreduce.jobhistor=
y.JobHistoryEventHandler: In stop, writing event AM_STARTED
> 2019-07-22 14:56:37,179 INFO [main] org.apache.hadoop.mapreduce.jobhistor=
y.JobHistoryEventHandler: Event Writer setup for JobId: job_1563805140414_0=
002, File: hdfs://hdfs-master-0.hdfs-service.hdfs:9000/tmp/hadoop-yarn/stag=
ing/root/.staging/job_1563805140414_0002/job_1563805140414_0002_2.jhist
> 2019-07-22 14:56:37,283 INFO [main] org.apache.hadoop.mapreduce.jobhistor=
y.JobHistoryEventHandler: In stop, writing event AM_STARTED
> 2019-07-22 14:56:37,284 INFO [main] org.apache.hadoop.mapreduce.jobhistor=
y.JobHistoryEventHandler: In stop, writing event JOB_SUBMITTED
> 2019-07-22 14:56:37,300 INFO [main] org.apache.hadoop.mapreduce.jobhistor=
y.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
> 2019-07-22 14:56:37,300 INFO [main] org.apache.hadoop.mapreduce.v2.app.MR=
AppMaster: Skipping cleaning up the staging dir. assuming AM will be retrie=
d.
> 2019-07-22 14:56:37,300 INFO [main] org.apache.hadoop.ipc.Server: Stoppin=
g server on 42017
> 2019-07-22 14:56:37,301 INFO [IPC Server listener on 42017] org.apache.ha=
doop.ipc.Server: Stopping IPC Server listener on 42017
> 2019-07-22 14:56:37,302 INFO [IPC Server Responder] org.apache.hadoop.ipc=
.Server: Stopping IPC Server Responder
> 2019-07-22 14:56:37,302 INFO [TaskHeartbeatHandler PingChecker] org.apach=
e.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread=
 interrupted
> 2019-07-22 14:56:37,308 INFO [Ping Checker] org.apache.hadoop.yarn.util.A=
bstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
> 2019-07-22 14:56:37,308 FATAL [main] org.apache.hadoop.mapreduce.v2.app.M=
RAppMaster: Error starting MRAppMaster
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop=
.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattem=
pt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:177)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.serviceStart(RMCommunicator.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAl=
locator.serviceStart(RMContainerAllocator.java:274)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$Cont=
ainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.CompositeService.serviceS=
tart(CompositeService.java:121)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serv=
iceStart(MRAppMaster.java:1272)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.service.AbstractService.start(Abs=
tractService.java:194)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.ru=
n(MRAppMaster.java:1723)
> =C2=A0=C2=A0 =C2=A0at java.security.AccessController.doPrivileged(Native =
Method)
> =C2=A0=C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.security.UserGroupInformation.doA=
s(UserGroupInformation.java:1893)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init=
AndStartAppMaster(MRAppMaster.java:1719)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main=
(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: I=
nvalid AMRMToken from appattempt_1563805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce0(Native Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce(NativeConstructorAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingConstructorAccessorImpl.newIn=
stance(DelegatingConstructorAccessorImpl.java:45)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Constructor.newInstance(Construct=
or.java:423)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateExcep=
tion(RPCUtil.java:53)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOExc=
eption(RPCUtil.java:80)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowEx=
ception(RPCUtil.java:119)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:109)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native=
 Method)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke(NativeM=
ethodAccessorImpl.java:62)
> =C2=A0=C2=A0 =C2=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del=
egatingMethodAccessorImpl.java:43)
> =C2=A0=C2=A0 =C2=A0at java.lang.reflect.Method.invoke(Method.java:498)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvokeMethod(RetryInvocationHandler.java:422)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeMethod(RetryInvocationHandler.java:165)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invoke(RetryInvocationHandler.java:157)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler$C=
all.invokeOnce(RetryInvocationHandler.java:95)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.io.retry.RetryInvocationHandler.i=
nvoke(RetryInvocationHandler.java:359)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy82.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicato=
r.register(RMCommunicator.java:160)
> =C2=A0=C2=A0 =C2=A0... 14 more
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.securi=
ty.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_156=
3805140414_0002_000002
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.getRpcResponse(Client.=
java:1507)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:227)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv=
oke(ProtobufRpcEngine.java:116)
> =C2=A0=C2=A0 =C2=A0at com.sun.proxy.$Proxy81.registerApplicationMaster(Un=
known Source)
> =C2=A0=C2=A0 =C2=A0at org.apache.hadoop.yarn.api.impl.pb.client.Applicati=
onMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterPro=
tocolPBClientImpl.java:106)
> =C2=A0=C2=A0 =C2=A0... 25 more
> 2019-07-22 14:56:37,309 INFO [main] org.apache.hadoop.util.ExitUtil: Exit=
ing with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: =
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMTo=
ken from appattempt_1563805140414_0002_000002
> {code}
> =C2=A0



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org