Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50ADC17CFC for ; Fri, 20 Feb 2015 02:10:04 +0000 (UTC) Received: (qmail 32169 invoked by uid 500); 20 Feb 2015 02:09:55 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 32054 invoked by uid 500); 20 Feb 2015 02:09:55 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 32044 invoked by uid 99); 20 Feb 2015 02:09:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Feb 2015 02:09:54 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of roland.depratti@cox.net designates 68.230.241.216 as permitted sender) Received: from [68.230.241.216] (HELO eastrmfepo201.cox.net) (68.230.241.216) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Feb 2015 02:09:07 +0000 Received: from eastrmimpo209 ([68.230.241.224]) by eastrmfepo201.cox.net (InterMail vM.8.01.05.15 201-2260-151-145-20131218) with ESMTP id <20150220020905.MEVC25062.eastrmfepo201.cox.net@eastrmimpo209> for ; Thu, 19 Feb 2015 21:09:05 -0500 Received: from MainPC ([72.195.141.1]) by eastrmimpo209 with cox id uS921p00r020h3401S93Ze; Thu, 19 Feb 2015 21:09:04 -0500 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020203.54E69741.0036,ss=1,re=0.000,fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.0 cv=H/cFNZki c=1 sm=1 a=w7pKjAbGsLCWhSjHPtj8rw==:17 a=kviXuzpPAAAA:8 a=vSGtT91vAAAA:8 a=lVBABN-2AAAA:8 a=Mzh-7l9-AAAA:8 a=mV9VRH-2AAAA:8 a=9fzzACetAAAA:8 a=DwiIph00AAAA:8 a=_Ii05PlEAAAA:8 a=X5jlqh1DsexSO__HtVcA:9 a=wPNLvfGTeEIA:10 a=ANQIFRr-Ju4A:10 a=bd5xfFXJ1iwA:10 a=Cyagqx0h8Z9FvRBw:21 a=BGISRsDDJWzfSe2g:21 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=LI4fRA_Hpb0NDd7_H3EA:9 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 a=wMJ1sLz-QqHSvsYa:21 a=5g8jWkpe-pifWmoo:21 a=w7pKjAbGsLCWhSjHPtj8rw==:117 X-CM-Score: 0.00 Authentication-Results: cox.net; auth=pass (LOGIN) smtp.auth=roland.depratti@cox.net From: "Roland DePratti" To: References: <54E65C2B.8070506@ulul.org> <54E65EBC.7050206@ulul.org> <00ba01d04caa$2410b390$6c321ab0$@cox.net> In-Reply-To: Subject: RE: Yarn AM is abending job when submitting a remote job to cluster Date: Thu, 19 Feb 2015 21:09:11 -0500 Message-ID: <00d701d04cb2$382e1c20$a88a5460$@cox.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_00D8_01D04C88.4F64E560" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQFQuGa6sOFAM//VUqp6c7kHOLNyRQExPsVlAfUey+0BEjRYqADPdDJmnc+y4DA= Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_00D8_01D04C88.4F64E560 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Xuan, =20 Thanks for asking. Here is the RM log. It almost looks like the log completes successfully (see red highlighting). =20 =20 =20 2015-02-19 19:55:43,315 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated = new applicationId: 12 2015-02-19 19:55:44,659 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: = Application with id 12 submitted by user cloudera 2015-02-19 19:55:44,659 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1424003606313_0012 2015-02-19 19:55:44,659 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera IP=3D192.168.2.185 OPERATION=3DSubmit Application Request TARGET=3DClientRMService RESULT=3DSUCCESS APPID=3Dapplication_1424003606313_0012 2015-02-19 19:55:44,659 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1424003606313_0012 State change from NEW to NEW_SAVING 2015-02-19 19:55:44,659 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: = Storing info for app: application_1424003606313_0012 2015-02-19 19:55:44,660 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1424003606313_0012 State change from NEW_SAVING to SUBMITTED 2015-02-19 19:55:44,666 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Accepted application application_1424003606313_0012 from user: cloudera, = in queue: default, currently num of applications: 1 2015-02-19 19:55:44,667 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1424003606313_0012 State change from SUBMITTED to ACCEPTED 2015-02-19 19:55:44,667 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1424003606313_0012_000001 2015-02-19 19:55:44,667 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from NEW to = SUBMITTED 2015-02-19 19:55:44,667 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Added Application Attempt appattempt_1424003606313_0012_000001 to = scheduler from user: cloudera 2015-02-19 19:55:44,669 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from SUBMITTED to SCHEDULED 2015-02-19 19:55:50,671 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from NEW = to ALLOCATED 2015-02-19 19:55:50,671 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera OPERATION=3DAM Allocated Container TARGET=3DSchedulerApp = RESULT=3DSUCCESS APPID=3Dapplication_1424003606313_0012 CONTAINERID=3Dcontainer_1424003606313_0012_01_000001 2015-02-19 19:55:50,671 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1424003606313_0012_01_000001 of capacity on host hadoop0.rdpratti.com:8041, which has 1 containers, used and available after allocation 2015-02-19 19:55:50,672 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erI nRM: Sending NMToken for nodeId : hadoop0.rdpratti.com:8041 for = container : container_1424003606313_0012_01_000001 2015-02-19 19:55:50,672 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from = ALLOCATED to ACQUIRED 2015-02-19 19:55:50,673 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erI nRM: Clear node set for appattempt_1424003606313_0012_000001 2015-02-19 19:55:50,673 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : Storing attempt: AppId: application_1424003606313_0012 AttemptId: appattempt_1424003606313_0012_000001 MasterContainer: Container: [ContainerId: container_1424003606313_0012_01_000001, NodeId: hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.2.253:8041 }, ] 2015-02-19 19:55:50,673 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from SCHEDULED to ALLOCATED_SAVING 2015-02-19 19:55:50,673 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from = ALLOCATED_SAVING to ALLOCATED 2015-02-19 19:55:50,673 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1424003606313_0012_000001 2015-02-19 19:55:50,674 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Setting up container Container: [ContainerId: container_1424003606313_0012_01_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM appattempt_1424003606313_0012_000001 2015-02-19 19:55:50,675 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Command to launch container container_1424003606313_0012_01_000001 : $JAVA_HOME/bin/java -Dlog4j.configuration=3Dcontainer-log4j.properties -Dyarn.app.container.log.dir=3D = -Dyarn.app.container.log.filesize=3D0 -Dhadoop.root.logger=3DINFO,CLA -Djava.net.preferIPv4Stack=3Dtrue = -Xmx209715200 org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout 2>/stderr=20 2015-02-19 19:55:50,675 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= age r: Create AMRMToken for ApplicationAttempt: appattempt_1424003606313_0012_000001 2015-02-19 19:55:50,675 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= age r: Creating password for appattempt_1424003606313_0012_000001 2015-02-19 19:55:50,688 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Done launching container Container: [ContainerId: container_1424003606313_0012_01_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM appattempt_1424003606313_0012_000001 2015-02-19 19:55:50,688 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from ALLOCATED to LAUNCHED 2015-02-19 19:55:50,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from = ACQUIRED to RUNNING 2015-02-19 19:55:57,941 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from = RUNNING to COMPLETED 2015-02-19 19:55:57,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt= : Completed container: container_1424003606313_0012_01_000001 in state: COMPLETED event:FINISHED 2015-02-19 19:55:57,942 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera OPERATION=3DAM Released Container TARGET=3DSchedulerApp = RESULT=3DSUCCESS APPID=3Dapplication_1424003606313_0012 CONTAINERID=3Dcontainer_1424003606313_0012_01_000001 2015-02-19 19:55:57,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1424003606313_0012_01_000001 of capacity on host hadoop0.rdpratti.com:8041, which = currently has 0 containers, used and available, release resources=3Dtrue 2015-02-19 19:55:57,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application attempt appattempt_1424003606313_0012_000001 released = container container_1424003606313_0012_01_000001 on node: host: hadoop0.rdpratti.com:8041 #containers=3D0 available=3D1457 used=3D0 with = event: FINISHED 2015-02-19 19:55:57,942 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : Updating application attempt appattempt_1424003606313_0012_000001 with final state: FAILED, and exit status: 1 2015-02-19 19:55:57,942 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from LAUNCHED to FINAL_SAVING 2015-02-19 19:55:57,942 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1424003606313_0012_000001 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= age r: Application finished, removing password for appattempt_1424003606313_0012_000001 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000001 State change from FINAL_SAVING to FAILED 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application appattempt_1424003606313_0012_000001 is done. = finalState=3DFAILED 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1424003606313_0012_000002 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo= : Application application_1424003606313_0012 requests cleared 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from NEW to = SUBMITTED 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Added Application Attempt appattempt_1424003606313_0012_000002 to = scheduler from user: cloudera 2015-02-19 19:55:57,943 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from SUBMITTED to SCHEDULED 2015-02-19 19:55:58,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Null container completed... 2015-02-19 19:56:03,950 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from NEW = to ALLOCATED 2015-02-19 19:56:03,950 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera OPERATION=3DAM Allocated Container TARGET=3DSchedulerApp = RESULT=3DSUCCESS APPID=3Dapplication_1424003606313_0012 CONTAINERID=3Dcontainer_1424003606313_0012_02_000001 2015-02-19 19:56:03,950 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1424003606313_0012_02_000001 of capacity on host hadoop0.rdpratti.com:8041, which has 1 containers, used and available after allocation 2015-02-19 19:56:03,950 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erI nRM: Sending NMToken for nodeId : hadoop0.rdpratti.com:8041 for = container : container_1424003606313_0012_02_000001 2015-02-19 19:56:03,951 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from = ALLOCATED to ACQUIRED 2015-02-19 19:56:03,951 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erI nRM: Clear node set for appattempt_1424003606313_0012_000002 2015-02-19 19:56:03,951 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : Storing attempt: AppId: application_1424003606313_0012 AttemptId: appattempt_1424003606313_0012_000002 MasterContainer: Container: [ContainerId: container_1424003606313_0012_02_000001, NodeId: hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.2.253:8041 }, ] 2015-02-19 19:56:03,952 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from SCHEDULED to ALLOCATED_SAVING 2015-02-19 19:56:03,952 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from = ALLOCATED_SAVING to ALLOCATED 2015-02-19 19:56:03,952 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1424003606313_0012_000002 2015-02-19 19:56:03,953 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Setting up container Container: [ContainerId: container_1424003606313_0012_02_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM appattempt_1424003606313_0012_000002 2015-02-19 19:56:03,953 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Command to launch container container_1424003606313_0012_02_000001 : $JAVA_HOME/bin/java -Dlog4j.configuration=3Dcontainer-log4j.properties -Dyarn.app.container.log.dir=3D = -Dyarn.app.container.log.filesize=3D0 -Dhadoop.root.logger=3DINFO,CLA -Djava.net.preferIPv4Stack=3Dtrue = -Xmx209715200 org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout 2>/stderr=20 2015-02-19 19:56:03,953 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= age r: Create AMRMToken for ApplicationAttempt: appattempt_1424003606313_0012_000002 2015-02-19 19:56:03,953 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= age r: Creating password for appattempt_1424003606313_0012_000002 2015-02-19 19:56:03,974 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Done launching container Container: [ContainerId: container_1424003606313_0012_02_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM appattempt_1424003606313_0012_000002 2015-02-19 19:56:03,974 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from ALLOCATED to LAUNCHED 2015-02-19 19:56:04,947 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from = ACQUIRED to RUNNING 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from = RUNNING to COMPLETED 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt= : Completed container: container_1424003606313_0012_02_000001 in state: COMPLETED event:FINISHED 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera OPERATION=3DAM Released Container TARGET=3DSchedulerApp = RESULT=3DSUCCESS APPID=3Dapplication_1424003606313_0012 CONTAINERID=3Dcontainer_1424003606313_0012_02_000001 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1424003606313_0012_02_000001 of capacity on host hadoop0.rdpratti.com:8041, which = currently has 0 containers, used and available, release resources=3Dtrue 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : Updating application attempt appattempt_1424003606313_0012_000002 with final state: FAILED, and exit status: 1 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application attempt appattempt_1424003606313_0012_000002 released = container container_1424003606313_0012_02_000001 on node: host: hadoop0.rdpratti.com:8041 #containers=3D0 available=3D1457 used=3D0 with = event: FINISHED 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from LAUNCHED to FINAL_SAVING 2015-02-19 19:56:10,956 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1424003606313_0012_000002 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= age r: Application finished, removing password for appattempt_1424003606313_0012_000002 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl : appattempt_1424003606313_0012_000002 State change from FINAL_SAVING to FAILED 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1424003606313_0012 with final state: FAILED 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1424003606313_0012 State change from ACCEPTED to = FINAL_SAVING 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1424003606313_0012 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application appattempt_1424003606313_0012_000002 is done. = finalState=3DFAILED 2015-02-19 19:56:10,957 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo= : Application application_1424003606313_0012 requests cleared 2015-02-19 19:56:10,990 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = Application application_1424003606313_0012 failed 2 times due to AM Container for appattempt_1424003606313_0012_000002 exited with exitCode: 1 due to: Exception from container-launch. Container id: container_1424003606313_0012_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=3D1:=20 at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)= at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch= Con tainer(DefaultContainerExecutor.java:197) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= ine rLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= ine rLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :11 45) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:6 15) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 2015-02-19 19:56:10,990 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1424003606313_0012 State change from FINAL_SAVING to FAILED 2015-02-19 19:56:10,991 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera OPERATION=3DApplication Finished - Failed TARGET=3DRMAppManager RESULT=3DFAILURE DESCRIPTION=3DApp failed with state: FAILED PERMISSIONS=3DApplication application_1424003606313_0012 failed 2 times = due to AM Container for appattempt_1424003606313_0012_000002 exited with = exitCode: 1 due to: Exception from container-launch. Container id: container_1424003606313_0012_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=3D1:=20 at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)= at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch= Con tainer(DefaultContainerExecutor.java:197) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= ine rLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= ine rLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :11 45) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:6 15) at java.lang.Thread.run(Thread.java:745) =20 =20 From: Xuan Gong [mailto:xgong@hortonworks.com]=20 Sent: Thursday, February 19, 2015 8:23 PM To: user@hadoop.apache.org Subject: Re: Yarn AM is abending job when submitting a remote job to = cluster =20 Hey, Roland: Could you also check the RM logs for this application, please ? = Maybe we could find something there. =20 Thanks =20 Xuan Gong =20 From: Roland DePratti Reply-To: "user@hadoop.apache.org" Date: Thursday, February 19, 2015 at 5:11 PM To: "user@hadoop.apache.org" Subject: RE: Yarn AM is abending job when submitting a remote job to = cluster =20 No, I hear you. =20 =20 I was just stating that the fact that hdfs works, there is something = right about the connectivity, that=92s all, i.e. Server is reachable, hadoop = was able to process the request =96 but like you said, doesn=92t mean yarn = works. =20 I tried both your solution and Alex=92s solution unfortunately without = any improvement. =20 Here is the command I am executing: =20 hadoop jar avgWordlength.jar solution.AvgWordLength -conf ~/conf/hadoop-cluster.xml /user/cloudera/shakespeare wordlength4 =20 Here is the new hadoop-cluseter.xml =20 fs.defaultFS hdfs://hadoop0.rdpratti.com:8020 mapreduce.jobtracker.address hadoop0.rdpratti.com:8032 yarn.resourcemanager.address hadoop0.rdpratti.com:8032 I also deleted the .staging directory under the submitting user. Plus restarted Job History Server.=20 =20 Resubmitted the job with the same result. Here is the log: =20 2015-02-19 19:56:05,061 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1424003606313_0012_000002 2015-02-19 19:56:05,468 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-02-19 19:56:05,471 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-02-19 19:56:05,471 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-02-19 19:56:05,473 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-02-19 19:56:05,476 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-02-19 19:56:05,490 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-02-19 19:56:05,621 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2015-02-19 19:56:05,621 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@3909f88f) 2015-02-19 19:56:05,684 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2015-02-19 19:56:05,923 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-02-19 19:56:05,925 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-02-19 19:56:05,929 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-02-19 19:56:05,930 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-02-19 19:56:05,934 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-02-19 19:56:05,958 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-02-19 19:56:06,529 WARN [main] = org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using = builtin-java classes where applicable 2015-02-19 19:56:06,719 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2015-02-19 19:56:06,837 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2015-02-19 19:56:06,881 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2015-02-19 19:56:06,882 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2015-02-19 19:56:06,882 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2015-02-19 19:56:06,883 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for = class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher= 2015-02-19 19:56:06,884 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2015-02-19 19:56:06,885 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for = class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2015-02-19 19:56:06,885 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for = class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2015-02-19 19:56:06,886 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType = for class = org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2015-02-19 19:56:06,899 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery is enabled. = Will try to recover from previous life on best effort basis. 2015-02-19 19:56:06,918 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is = at hdfs://hadoop0.rdpratti.com:8020/user/cloudera/.staging/job_1424003606313= _00 12/job_1424003606313_0012_1.jhist 2015-02-19 19:56:07,377 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Read completed tasks = from history 0 2015-02-19 19:56:07,423 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for = class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2015-02-19 19:56:07,453 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2015-02-19 19:56:07,507 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot = period at 10 second(s). 2015-02-19 19:56:07,507 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2015-02-19 19:56:07,515 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token = for job_1424003606313_0012 to jobTokenSecretManager 2015-02-19 19:56:07,536 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1424003606313_0012 because: not enabled; too much RAM; 2015-02-19 19:56:07,555 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1424003606313_0012 =3D 5343207. Number of splits =3D 5 2015-02-19 19:56:07,557 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces = for job job_1424003606313_0012 =3D 1 2015-02-19 19:56:07,557 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1424003606313_0012Job Transitioned from NEW to INITED 2015-02-19 19:56:07,558 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1424003606313_0012. 2015-02-19 19:56:07,618 INFO [main] = org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-02-19 19:56:07,630 INFO [Socket Reader #1 for port 46841] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46841 2015-02-19 19:56:07,648 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the = server 2015-02-19 19:56:07,648 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-02-19 19:56:07,649 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at hadoop0.rdpratti.com/192.168.2.253:46841 2015-02-19 19:56:07,650 INFO [IPC Server listener on 46841] org.apache.hadoop.ipc.Server: IPC Server listener on 46841: starting 2015-02-19 19:56:07,721 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-02-19 19:56:07,727 INFO [main] = org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2015-02-19 19:56:07,739 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=3Dorg.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2015-02-19 19:56:07,745 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=3Dorg.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2015-02-19 19:56:07,745 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=3Dorg.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2015-02-19 19:56:07,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2015-02-19 19:56:07,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2015-02-19 19:56:07,760 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 39939 2015-02-19 19:56:07,760 INFO [main] org.mortbay.log: = jetty-6.1.26.cloudera.4 2015-02-19 19:56:07,789 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/hadoop-yar= n-c ommon-2.5.0-cdh5.3.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_39939_mapreduce____.o5qk0w/webapp 2015-02-19 19:56:08,156 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:39939 2015-02-19 19:56:08,157 INFO [main] = org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 39939 2015-02-19 19:56:08,629 INFO [main] = org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2015-02-19 19:56:08,634 INFO [main] = org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-02-19 19:56:08,635 INFO [Socket Reader #1 for port 43858] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43858 2015-02-19 19:56:08,639 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-02-19 19:56:08,642 INFO [IPC Server listener on 43858] org.apache.hadoop.ipc.Server: IPC Server listener on 43858: starting 2015-02-19 19:56:08,663 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2015-02-19 19:56:08,663 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2015-02-19 19:56:08,663 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2015-02-19 19:56:08,797 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-02-19 19:56:08,798 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-02-19 19:56:08,798 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-02-19 19:56:08,798 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-02-19 19:56:08,799 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-02-19 19:56:08,809 WARN [main] = org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-02-19 19:56:08,821 INFO [main] = org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/192.168.2.185:8030 2015-02-19 19:56:08,975 WARN [main] org.apache.hadoop.security.UserGroupInformation: = PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.to= ken .SecretManager$InvalidToken): appattempt_1424003606313_0012_000002 not = found in AMRMTokenSecretManager. 2015-02-19 19:56:08,976 WARN [main] org.apache.hadoop.ipc.Client: = Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Se= cre tManager$InvalidToken): appattempt_1424003606313_0012_000002 not found = in AMRMTokenSecretManager. 2015-02-19 19:56:08,976 WARN [main] org.apache.hadoop.security.UserGroupInformation: = PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.to= ken .SecretManager$InvalidToken): appattempt_1424003606313_0012_000002 not = found in AMRMTokenSecretManager. 2015-02-19 19:56:08,981 ERROR [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception = while registering org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1424003606313_0012_000002 not found in = AMRMTokenSecretManager. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc= ces sorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst= ruc torAccessorImpl.java:45) at = java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:1= 04) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie= ntI mpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:= 109 ) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :57 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl .java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc= ati onHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH= and ler.java:102) at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown = Source) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunica= tor .java:161) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommu= nic ator.java:122) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(R= MCo ntainerAllocator.java:238) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)= at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.s= erv iceStart(MRAppMaster.java:807) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)= at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.= jav a:120) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.j= ava :1075) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)= at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:147= 8) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .ja va:1642) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAp= pMa ster.java:1474) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407= ) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Se= cre tManager$InvalidToken): appattempt_1424003606313_0012_000002 not found = in AMRMTokenSecretManager. at org.apache.hadoop.ipc.Client.call(Client.java:1411) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.= jav a:206) at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown = Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie= ntI mpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:= 106 ) ... 22 more 2015-02-19 19:56:08,983 INFO [main] org.apache.hadoop.service.AbstractService: Service RMCommunicator failed = in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1424003606313_0012_000002 not found in = AMRMTokenSecretManager. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1424003606313_0012_000002 not found in = AMRMTokenSecretManager. =20 =20 =20 =20 From: Ulul [mailto:hadoop@ulul.org]=20 Sent: Thursday, February 19, 2015 5:08 PM To: user@hadoop.apache.org Subject: Re: Yarn AM is abending job when submitting a remote job to = cluster =20 Is your point is that using the hdfs:// prefix is valid since our hdfs client works ? fs.defaultFS defines the namenode address and the filesystem type. It = doen't imply that the prefix should be used for yarn and mapreduce options that = are not directly linked to hdfs=20 Le 19/02/2015 22:56, Ulul a =E9crit : In that case it's just between your hdfs client, the NN and the DNs, no = YARN or MR component involved. The fact that this works is not related to your MR job not succeeding. Le 19/02/2015 22:45, roland.depratti a =E9crit : Thanks for looking at my problem. =20 I can run an hdfs command from the client, with the config file listed, = that does a cat on a file in hdfs on the remote cluster and returns the = contents of that file to the client. =20 - rd =20 =20 Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Ulul =20 Date:02/19/2015 4:03 PM (GMT-05:00)=20 To: user@hadoop.apache.org=20 Subject: Re: Yarn AM is abending job when submitting a remote job to = cluster Hi Doesn't seem like an ssl error to me (the log states that attempts to=20 override final properties are ignored) On the other hand the configuration seems wrong=20 :mapreduce.jobtracker.address and yarn.resourcemanager.address should=20 only contain an IP or a hostname. You should remove 'hdfs://' though the = log doesn't suggest it has anything to do with your problem.... And what do you mean by an "HDFS job" ? Ulul Le 19/02/2015 04:22, daemeon reiydelle a =E9crit : > I would guess you do not have your ssl certs set up, client or server, = > based on the error. > > *** > ....... > ***=93Life should not be a journey to the grave with the intention of=20 > arriving safely in a > pretty and well preserved body, but rather to skid in broadside in a=20 > cloud of smoke, > thoroughly used up, totally worn out, and loudly proclaiming =93Wow!=20 > What a Ride!=94* > - Hunter Thompson > > Daemeon C.M. Reiydelle > USA (+1) 415.501.0198 > London (+44) (0) 20 8144 9872*/ > / > > On Wed, Feb 18, 2015 at 5:19 PM, Roland DePratti=20 > > wrote: > > I have been searching for a handle on a problem without very > little clues. Any help pointing me to the right direction will be > huge. > > I have not received any input form the Cloudera google groups. > Perhaps this is more Yarn based and I am hoping I have more luck = here. > > Any help is greatly appreciated. > > I am running a Hadoop cluster using CDH5.3. I also have a client > machine with a standalone one node setup (VM). > > All environments are running CentOS 6.6. > > I have submitted some Java mapreduce jobs locally on both the > cluster and the standalone environment with successfully = completions. > > I can submit a remote HDFS job from client to cluster using -conf > hadoop-cluster.xml (see below) and get data back from the cluster > with no problem. > > When submitted remotely the mapreduce jobs remotely, I get an AM > error: > > AM fails the job with the error: > > > SecretManager$InvalidToken: > appattempt_1424003606313_0001_000002 not found in > AMRMTokenSecretManager > > > I searched /var/log/secure on the client and cluster with no > unusual messages. > > Here is the contents of hadoop-cluster.xml: > > > > > > > fs.defaultFS > hdfs://mycluser:8020 > > > mapreduce.jobtracker.address > hdfs://mycluster:8032 > > > yarn.resourcemanager.address > hdfs://mycluster:8032 > > > Here is the output from the job log on the cluster: > > 2015-02-15 07:51:06,544 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created > MRAppMaster for application appattempt_1424003606313_0001_000002 > > 2015-02-15 07:51:06,949 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.require.client.cert; = Ignoring. > > 2015-02-15 07:51:06,952 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > > 2015-02-15 07:51:06,952 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.client.conf; Ignoring. > > 2015-02-15 07:51:06,954 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.keystores.factory.class;=20 > Ignoring. > > 2015-02-15 07:51:06,957 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.server.conf; Ignoring. > > 2015-02-15 07:51:06,973 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > > 2015-02-15 07:51:07,241 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with = tokens: > > 2015-02-15 07:51:07,241 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: > YARN_AM_RM_TOKEN, Service: , Ident: > (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33be1aa0) > > 2015-02-15 07:51:07,332 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred > newApiCommitter. > > 2015-02-15 07:51:07,627 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.require.client.cert; = Ignoring. > > 2015-02-15 07:51:07,632 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > > 2015-02-15 07:51:07,632 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.client.conf; Ignoring. > > 2015-02-15 07:51:07,639 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.keystores.factory.class;=20 > Ignoring. > > 2015-02-15 07:51:07,645 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: hadoop.ssl.server.conf; Ignoring. > > 2015-02-15 07:51:07,663 WARN [main] > org.apache.hadoop.conf.Configuration: job.xml:an attempt to > override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > > 2015-02-15 07:51:08,237 WARN [main] > org.apache.hadoop.util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java > classes where applicable > > 2015-02-15 07:51:08,429 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter > set in config null > > 2015-02-15 07:51:08,499 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter > > 2015-02-15 07:51:08,526 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.jobhistory.EventType for class > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler > > 2015-02-15 07:51:08,527 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for > class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher > > 2015-02-15 07:51:08,561 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for > class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher > > 2015-02-15 07:51:08,562 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType > for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher= > > 2015-02-15 07:51:08,566 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for > class = org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler > > 2015-02-15 07:51:08,568 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType > for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher > > 2015-02-15 07:51:08,568 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType > for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter > > 2015-02-15 07:51:08,570 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType > for class > = org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter > > 2015-02-15 07:51:08,599 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery is > enabled. Will try to recover from previous life on best effort = basis. > > 2015-02-15 07:51:08,642 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history > file is at > hdfs://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/j= ob_ 1424003606313_0001_1.jhist > > > _2015-02-15 > _07:51:09,147 > INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Read > completed tasks from history 0 > > 2015-02-15 07:51:09,193 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type > for class > = org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler > > 2015-02-15 07:51:09,222 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties > from hadoop-metrics2.properties > > 2015-02-15 07:51:09,277 INFO [main] =20 =20 ------=_NextPart_000_00D8_01D04C88.4F64E560 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Xuan,

 

Thanks for asking. Here is the RM log. It almost looks like the log = completes successfully (see red highlighting).

 

 

 

2015-02-19 = 19:55:43,315 INFO = org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated = new applicationId: 12
2015-02-19 19:55:44,659 INFO = org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: = Application with id 12 submitted by user cloudera
2015-02-19 = 19:55:44,659 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing = application with id application_1424003606313_0012
2015-02-19 = 19:55:44,659 INFO = org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera    IP=3D192.168.2.185    = OPERATION=3DSubmit Application Request    = TARGET=3DClientRMService    = RESULT=3DSUCCESS    = APPID=3Dapplication_1424003606313_0012
2015-02-19 19:55:44,659 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = application_1424003606313_0012 State change from NEW to = NEW_SAVING
2015-02-19 19:55:44,659 INFO = org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: = Storing info for app: application_1424003606313_0012
2015-02-19 = 19:55:44,660 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = application_1424003606313_0012 State change from NEW_SAVING to = SUBMITTED
2015-02-19 19:55:44,666 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Accepted application application_1424003606313_0012 from user: = cloudera, in queue: default, currently num of applications: = 1
2015-02-19 19:55:44,667 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = application_1424003606313_0012 State change from SUBMITTED to = ACCEPTED
2015-02-19 19:55:44,667 INFO = org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: = Registering app attempt : = appattempt_1424003606313_0012_000001
2015-02-19 19:55:44,667 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from NEW to = SUBMITTED
2015-02-19 19:55:44,667 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Added Application Attempt appattempt_1424003606313_0012_000001 to = scheduler from user: cloudera
2015-02-19 19:55:44,669 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from SUBMITTED to = SCHEDULED
2015-02-19 19:55:50,671 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from NEW = to ALLOCATED
2015-02-19 19:55:50,671 INFO = org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera    OPERATION=3DAM Allocated = Container    TARGET=3DSchedulerApp    = RESULT=3DSUCCESS    = APPID=3Dapplication_1424003606313_0012    = CONTAINERID=3Dcontainer_1424003606313_0012_01_000001
2015-02-19 = 19:55:50,671 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: = Assigned container container_1424003606313_0012_01_000001 of capacity = <memory:1024, vCores:1> on host hadoop0.rdpratti.com:8041, which = has 1 containers, <memory:1024, vCores:1> used and <memory:433, = vCores:1> available after allocation
2015-02-19 19:55:50,672 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erInRM: Sending NMToken for nodeId : hadoop0.rdpratti.com:8041 for = container : container_1424003606313_0012_01_000001
2015-02-19 = 19:55:50,672 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from = ALLOCATED to ACQUIRED
2015-02-19 19:55:50,673 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erInRM: Clear node set for = appattempt_1424003606313_0012_000001
2015-02-19 19:55:50,673 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: Storing attempt: AppId: application_1424003606313_0012 AttemptId: = appattempt_1424003606313_0012_000001 MasterContainer: Container: = [ContainerId: container_1424003606313_0012_01_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, = Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { = kind: ContainerToken, service: 192.168.2.253:8041 }, ]
2015-02-19 = 19:55:50,673 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from SCHEDULED to = ALLOCATED_SAVING
2015-02-19 19:55:50,673 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from = ALLOCATED_SAVING to ALLOCATED
2015-02-19 19:55:50,673 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Launching masterappattempt_1424003606313_0012_000001
2015-02-19 = 19:55:50,674 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Setting up container Container: [ContainerId: = container_1424003606313_0012_01_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, = Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { = kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM = appattempt_1424003606313_0012_000001
2015-02-19 19:55:50,675 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Command to launch container container_1424003606313_0012_01_000001 : = $JAVA_HOME/bin/java -Dlog4j.configuration=3Dcontainer-log4j.properties = -Dyarn.app.container.log.dir=3D<LOG_DIR> = -Dyarn.app.container.log.filesize=3D0 = -Dhadoop.root.logger=3DINFO,CLA  -Djava.net.preferIPv4Stack=3Dtrue = -Xmx209715200 org.apache.hadoop.mapreduce.v2.app.MRAppMaster = 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2015-02-19 = 19:55:50,675 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= ager: Create AMRMToken for ApplicationAttempt: = appattempt_1424003606313_0012_000001
2015-02-19 19:55:50,675 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= ager: Creating password for = appattempt_1424003606313_0012_000001
2015-02-19 19:55:50,688 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Done launching container Container: [ContainerId: = container_1424003606313_0012_01_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, = Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { = kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM = appattempt_1424003606313_0012_000001
20= 15-02-19 19:55:50,688 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from ALLOCATED to = LAUNCHED
2015-02-19 19:55:50,928 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from = ACQUIRED to RUNNING
2015-02-19 19:55:57,941 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_01_000001 Container Transitioned from = RUNNING to COMPLETED
2015-02-19 19:55:57,941 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt= : Completed container: container_1424003606313_0012_01_000001 in state: = COMPLETED event:FINISHED
2015-02-19 19:55:57,942 INFO = org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera    OPERATION=3DAM Released = Container    TARGET=3DSchedulerApp    = RESULT=3DSUCCESS    = APPID=3Dapplication_1424003606313_0012    = CONTAINERID=3Dcontainer_1424003606313_0012_01_000001
2015-02-19 = 19:55:57,942 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: = Released container container_1424003606313_0012_01_000001 of capacity = <memory:1024, vCores:1> on host hadoop0.rdpratti.com:8041, which = currently has 0 containers, <memory:0, vCores:0> used and = <memory:1= 457, vCores:2> available, release resources=3Dtrue
2015-02-19 = 19:55:57,942 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application attempt appattempt_1424003606313_0012_000001 released = container container_1424003606313_0012_01_000001 on node: host: = hadoop0.rdpratti.com:8041 #containers=3D0 available=3D1457 used=3D0 with = event: FINISHED
2015-02-19 19:55:57,942 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: Updating application attempt appattempt_1424003606313_0012_000001 = with final state: FAILED, and exit status: 1
2015-02-19 19:55:57,942 = INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from LAUNCHED to = FINAL_SAVING
2015-02-19 19:55:57,942 INFO = org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: = Unregistering app attempt : = appattempt_1424003606313_0012_000001
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= ager: Application finished, removing password for = appattempt_1424003606313_0012_000001
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000001 State change from FINAL_SAVING = to FAILED
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application appattempt_1424003606313_0012_000001 is done. = finalState=3DFAILED
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: = Registering app attempt : = appattempt_1424003606313_0012_000002
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo= : Application application_1424003606313_0012 requests = cleared
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from NEW to = SUBMITTED
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Added Application Attempt appattempt_1424003606313_0012_000002 to = scheduler from user: cloudera
2015-02-19 19:55:57,943 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from SUBMITTED to = SCHEDULED
2015-02-19 19:55:58,941 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Null container completed...
2015-02-19 19:56:03,950 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from NEW = to ALLOCATED
2015-02-19 19:56:03,950 INFO = org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera    OPERATION=3DAM Allocated = Container    TARGET=3DSchedulerApp    = RESULT=3DSUCCESS    = APPID=3Dapplication_1424003606313_0012    = CONTAINERID=3Dcontainer_1424003606313_0012_02_000001
2015-02-19 = 19:56:03,950 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: = Assigned container container_1424003606313_0012_02_000001 of capacity = <memory:1024, vCores:1> on host hadoop0.rdpratti.com:8041, which = has 1 containers, <memory:1024, vCores:1> used and <memory:433, = vCores:1> available after allocation
2015-02-19 19:56:03,950 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erInRM: Sending NMToken for nodeId : hadoop0.rdpratti.com:8041 for = container : container_1424003606313_0012_02_000001
2015-02-19 = 19:56:03,951 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from = ALLOCATED to ACQUIRED
2015-02-19 19:56:03,951 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManag= erInRM: Clear node set for = appattempt_1424003606313_0012_000002
2015-02-19 19:56:03,951 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: Storing attempt: AppId: application_1424003606313_0012 AttemptId: = appattempt_1424003606313_0012_000002 MasterContainer: Container: = [ContainerId: container_1424003606313_0012_02_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, = Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { = kind: ContainerToken, service: 192.168.2.253:8041 }, ]
2015-02-19 = 19:56:03,952 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from SCHEDULED to = ALLOCATED_SAVING
2015-02-19 19:56:03,952 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from = ALLOCATED_SAVING to ALLOCATED
2015-02-19 19:56:03,952 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Launching masterappattempt_1424003606313_0012_000002
2015-02-19 = 19:56:03,953 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Setting up container Container: [ContainerId: = container_1424003606313_0012_02_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, = Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { = kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM = appattempt_1424003606313_0012_000002
2015-02-19 19:56:03,953 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Command to launch container container_1424003606313_0012_02_000001 : = $JAVA_HOME/bin/java -Dlog4j.configuration=3Dcontainer-log4j.properties = -Dyarn.app.container.log.dir=3D<LOG_DIR> = -Dyarn.app.container.log.filesize=3D0 = -Dhadoop.root.logger=3DINFO,CLA  -Djava.net.preferIPv4Stack=3Dtrue = -Xmx209715200 org.apache.hadoop.mapreduce.v2.app.MRAppMaster = 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2015-02-19 = 19:56:03,953 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= ager: Create AMRMToken for ApplicationAttempt: = appattempt_1424003606313_0012_000002
2015-02-19 19:56:03,953 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= ager: Creating password for = appattempt_1424003606313_0012_000002
2015-02-19 19:56:03,974 INFO = org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: = Done launching container Container: [ContainerId: = container_1424003606313_0012_02_000001, NodeId: = hadoop0.rdpratti.com:8041, NodeHttpAddress: hadoop0.rdpratti.com:8042, = Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { = kind: ContainerToken, service: 192.168.2.253:8041 }, ] for AM = appattempt_1424003606313_0012_000002
2015-02-19 19:56:03,974 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from ALLOCATED to = LAUNCHED
2015-02-19 19:56:04,947 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from = ACQUIRED to RUNNING
2015-02-19 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl= : container_1424003606313_0012_02_000001 Container Transitioned from = RUNNING to COMPLETED
2015-02-19 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt= : Completed container: container_1424003606313_0012_02_000001 in state: = COMPLETED event:FINISHED
2015-02-19 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera    OPERATION=3DAM Released = Container    TARGET=3DSchedulerApp    = RESULT=3DSUCCESS    = APPID=3Dapplication_1424003606313_0012    = CONTAINERID=3Dcontainer_1424003606313_0012_02_000001
2015-02-19 = 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: = Released container container_1424003606313_0012_02_000001 of capacity = <memory:1024, vCores:1> on host hadoop0.rdpratti.com:8041, which = currently has 0 containers, <memory:0, vCores:0> used and = <memory:1457, vCores:2> available, release = resources=3Dtrue
2015-02-19 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: Updating application attempt appattempt_1424003606313_0012_000002 = with final state: FAILED, and exit status: 1
2015-02-19 19:56:10,956 = INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application attempt appattempt_1424003606313_0012_000002 released = container container_1424003606313_0012_02_000001 on node: host: = hadoop0.rdpratti.com:8041 #containers=3D0 available=3D1457 used=3D0 with = event: FINISHED
2015-02-19 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from LAUNCHED to = FINAL_SAVING
2015-02-19 19:56:10,956 INFO = org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: = Unregistering app attempt : = appattempt_1424003606313_0012_000002
2015-02-19 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretMan= ager: Application finished, removing password for = appattempt_1424003606313_0012_000002
2015-02-19 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptI= mpl: appattempt_1424003606313_0012_000002 State change from FINAL_SAVING = to FAILED
2015-02-19 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating = application application_1424003606313_0012 with final state: = FAILED
2015-02-19 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = application_1424003606313_0012 State change from ACCEPTED to = FINAL_SAVING
2015-02-19 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: = Updating info for app: application_1424003606313_0012
2015-02-19 = 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule= r: Application appattempt_1424003606313_0012_000002 is done. = finalState=3DFAILED
2015-02-19 19:56:10,957 INFO = org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo= : Application application_1424003606313_0012 requests = cleared
2015-02-19 19:56:10,990 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = Application application_1424003606313_0012 failed 2 times due to AM = Container for appattempt_1424003606313_0012_000002 exited with  = exitCode: 1 due to: Exception from container-launch.
Container id: = container_1424003606313_0012_02_000001
Exit code: 1
Stack trace: = ExitCodeException exitCode=3D1:
    at = org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
  &n= bsp; at = org.apache.hadoop.util.Shell.run(Shell.java:455)
    = at = org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)=
    at = org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch= Container(DefaultContainerExecutor.java:197)
    at = org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= inerLaunch.call(ContainerLaunch.java:299)
    at = org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= inerLaunch.call(ContainerLaunch.java:81)
    at = java.util.concurrent.FutureTask.run(FutureTask.java:262)
  &= nbsp; at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1145)
    at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:615)
    at = java.lang.Thread.run(Thread.java:745)


Container exited with a = non-zero exit code 1
.Failing this attempt.. Failing the = application.
2015-02-19 19:56:10,990 INFO = org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: = application_1424003606313_0012 State change from FINAL_SAVING to = FAILED
2015-02-19 19:56:10,991 WARN = org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: = USER=3Dcloudera    OPERATION=3DApplication Finished - = Failed    TARGET=3DRMAppManager    = RESULT=3DFAILURE    DESCRIPTION=3DApp failed with state: = FAILED    PERMISSIONS=3DApplication = application_1424003606313_0012 failed 2 times due to AM Container for = appattempt_1424003606313_0012_000002 exited with  exitCode: 1 due = to: Exception from container-launch.
Container id: = container_1424003606313_0012_02_000001
Exit code: 1
Stack trace: = ExitCodeException exitCode=3D1:
    at = org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
  &n= bsp; at = org.apache.hadoop.util.Shell.run(Shell.java:455)
    = at = org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)=
    at = org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch= Container(DefaultContainerExecutor.java:197)
    at = org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= inerLaunch.call(ContainerLaunch.java:299)
    at = org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= inerLaunch.call(ContainerLaunch.java:81)
    at = java.util.concurrent.FutureTask.run(FutureTask.java:262)
  &= nbsp; at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1145)
    at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:615)
    at = java.lang.Thread.run(Thread.java:745)

 

 

From: Xuan Gong [mailto:xgong@hortonworks.com]
Sent: = Thursday, February 19, 2015 8:23 PM
To: = user@hadoop.apache.org
Subject: Re: Yarn AM is abending job = when submitting a remote job to = cluster

 

Hey, <= /span>Roland:=

  =   Could you also check the RM logs for this application, please ? = Maybe we could find something there.

 =

Thanks=

 =

Xuan = Gong

 =

From: = Roland = DePratti <roland.depratti@cox.net>Reply-To: "user@hadoop.apache.org" = <user@hadoop.apache.org>
= Date: Thursday, February 19, 2015 at 5:11 PM
To: = "user@hadoop.apache.org" = <user@hadoop.apache.org>
= Subject: RE: Yarn AM is abending job when submitting a remote job = to cluster

 =

No, I hear you. 

 

I was just stating that the fact that hdfs works, there is something = right about the connectivity, that’s all, i.e. Server is = reachable, hadoop was able to process the request – but like you = said, doesn’t mean yarn works.

 

I tried both your solution and Alex’s solution unfortunately = without any improvement.

 

Here is the command I am executing:

 

hadoop jar avgWordlength.jar  solution.AvgWordLength -conf = ~/conf/hadoop-cluster.xml /user/cloudera/shakespeare = wordlength4

 

Here is the new hadoop-cluseter.xml

 

<?xml = version=3D"1.0" = encoding=3D"UTF-8"?>

<!--generated by = Roland-->
<configuration>
  = <property>
    = <name>fs.defaultFS</name>
    = <value>hdfs://hadoop0.rdpratti.com:8020</value>
  = </property>
  <property>
    = <name>mapreduce.jobtracker.address</name>
  &nbs= p; <value>hadoop0.rdpratti.com:8032</value>
  = </property>
  <property>
    = <name>yarn.resourcemanager.address</name>
  &nbs= p; <value>hadoop0.rdpratti.com:8032</value>
  = </property>


I also deleted the .staging directory under the submitting user. Plus = restarted Job History Server.

 

Resubmitted the job with the same result. Here is the = log:

 

2015-02-19 19:56:05,061 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for =
application =
appattempt_1424003606313_0012_000002
2015-02-19 19:56:05,468 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.require.client.cert;  =
Ignoring.
2015-02-19 19:56:05,471 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: =
mapreduce.job.end-notification.max.retry.interval;  =
Ignoring.
2015-02-19 19:56:05,471 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.client.conf;  =
Ignoring.
2015-02-19 19:56:05,473 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.keystores.factory.class;  =
Ignoring.
2015-02-19 19:56:05,476 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.server.conf;  =
Ignoring.
2015-02-19 19:56:05,490 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: mapreduce.job.end-notification.max.attempts;  =
Ignoring.
2015-02-19 19:56:05,621 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with =
tokens:
2015-02-19 19:56:05,621 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, =
Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@3909f88f)
2015-02-19 =
19:56:05,684 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: =
Using mapred newApiCommitter.
2015-02-19 19:56:05,923 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.require.client.cert;  =
Ignoring.
2015-02-19 19:56:05,925 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: =
mapreduce.job.end-notification.max.retry.interval;  =
Ignoring.
2015-02-19 19:56:05,929 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.client.conf;  =
Ignoring.
2015-02-19 19:56:05,930 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.keystores.factory.class;  =
Ignoring.
2015-02-19 19:56:05,934 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.server.conf;  =
Ignoring.
2015-02-19 19:56:05,958 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: mapreduce.job.end-notification.max.attempts;  =
Ignoring.
2015-02-19 19:56:06,529 WARN [main] =
org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop =
library for your platform... using builtin-java classes where =
applicable
2015-02-19 19:56:06,719 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in =
config null
2015-02-19 19:56:06,837 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is =
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2015-02-19 19:56:06,881 INFO =
[main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class =
org.apache.hadoop.mapreduce.jobhistory.EventType for class =
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler<=
/span>
2015-02-19 19:56:06,882 =
INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for =
class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2015-02-19 19:56:06,882 =
INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for =
class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2015-02-19 =
19:56:06,883 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: =
Registering class =
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for =
class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher=

2015-02-19 =
19:56:06,884 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: =
Registering class =
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class =
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2015-02-19 19:56:06,885 =
INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType =
for class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher<=
o:p>
2015-02-19 =
19:56:06,885 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: =
Registering class =
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for =
class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-02-19 =
19:56:06,886 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: =
Registering class =
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType =
for class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2015-02-19 =
19:56:06,899 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: =
Recovery is enabled. Will try to recover from previous life on best =
effort basis.
2015-02-19 19:56:06,918 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is =
at =
hdfs://hadoop0.rdpratti.com:8020/user/cloudera/.staging/job_1424003606313=
_0012/job_1424003606313_0012_1.jhist
2015-02-19 19:56:07,377 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Read completed tasks =
from history 0
2015-02-19 19:56:07,423 INFO [main] =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class =
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for =
class =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler=

2015-02-19 =
19:56:07,453 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: =
loaded properties from =
hadoop-metrics2.properties
2015-02-19 19:56:07,507 INFO [main] =
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot =
period at 10 second(s).
2015-02-19 19:56:07,507 INFO [main] =
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics =
system started
2015-02-19 19:56:07,515 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token =
for job_1424003606313_0012 to =
jobTokenSecretManager
2015-02-19 19:56:07,536 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing =
job_1424003606313_0012 because: not enabled; too much =
RAM;
2015-02-19 =
19:56:07,555 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job =
job_1424003606313_0012 =3D 5343207. Number of splits =3D =
5
2015-02-19 =
19:56:07,557 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces =
for job job_1424003606313_0012 =3D 1
2015-02-19 19:56:07,557 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: =
job_1424003606313_0012Job Transitioned from NEW to =
INITED
2015-02-19 19:56:07,558 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching =
normal, non-uberized, multi-container job =
job_1424003606313_0012.
2015-02-19 19:56:07,618 INFO [main] =
org.apache.hadoop.ipc.CallQueueManager: Using callQueue class =
java.util.concurrent.LinkedBlockingQueue
2015-02-19 19:56:07,630 INFO [Socket Reader #1 =
for port 46841] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 =
for port 46841
2015-02-19 19:56:07,648 INFO [main] =
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding =
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the =
server
2015-02-19 19:56:07,648 INFO [IPC Server =
Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: =
starting
2015-02-19 19:56:07,649 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated =
MRClientService at =
hadoop0.rdpratti.com/192.168.2.253:46841
2015-02-19 19:56:07,650 INFO [IPC Server =
listener on 46841] org.apache.hadoop.ipc.Server: IPC Server listener on =
46841: starting
2015-02-19 19:56:07,721 INFO [main] =
org.mortbay.log: Logging to =
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via =
org.mortbay.log.Slf4jLog
2015-02-19 19:56:07,727 INFO [main] =
org.apache.hadoop.http.HttpRequestLog: Http request log for =
http.requests.mapreduce is not defined
2015-02-19 19:56:07,739 INFO [main] =
org.apache.hadoop.http.HttpServer2: Added global filter 'safety' =
(class=3Dorg.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-02-19 19:56:07,745 =
INFO [main] org.apache.hadoop.http.HttpServer2: Added filter =
AM_PROXY_FILTER =
(class=3Dorg.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to =
context mapreduce
2015-02-19 19:56:07,745 INFO [main] =
org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER =
(class=3Dorg.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to =
context static
2015-02-19 19:56:07,749 INFO [main] =
org.apache.hadoop.http.HttpServer2: adding path spec: =
/mapreduce/*
2015-02-19 19:56:07,749 INFO [main] =
org.apache.hadoop.http.HttpServer2: adding path spec: =
/ws/*
2015-02-19 =
19:56:07,760 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound =
to port 39939
2015-02-19 19:56:07,760 INFO [main] =
org.mortbay.log: =
jetty-6.1.26.cloudera.4
2015-02-19 19:56:07,789 INFO [main] =
org.mortbay.log: Extract =
jar:file:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/hadoop-yar=
n-common-2.5.0-cdh5.3.0.jar!/webapps/mapreduce to =
/tmp/Jetty_0_0_0_0_39939_mapreduce____.o5qk0w/webapp
2015-02-19 19:56:08,156 INFO [main] =
org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:39939
2015-02-19 =
19:56:08,157 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app =
/mapreduce started at 39939
2015-02-19 19:56:08,629 INFO [main] =
org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice =
modules
2015-02-19 19:56:08,634 INFO [main] =
org.apache.hadoop.ipc.CallQueueManager: Using callQueue class =
java.util.concurrent.LinkedBlockingQueue
2015-02-19 19:56:08,635 INFO [Socket Reader #1 =
for port 43858] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 =
for port 43858
2015-02-19 19:56:08,639 INFO [IPC Server =
Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: =
starting
2015-02-19 19:56:08,642 INFO [IPC Server listener =
on 43858] org.apache.hadoop.ipc.Server: IPC Server listener on 43858: =
starting
2015-02-19 19:56:08,663 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: =
nodeBlacklistingEnabled:true
2015-02-19 19:56:08,663 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: =
maxTaskFailuresPerNode is 3
2015-02-19 19:56:08,663 INFO [main] =
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: =
blacklistDisablePercent is 33
2015-02-19 19:56:08,797 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.require.client.cert;  =
Ignoring.
2015-02-19 19:56:08,798 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: =
mapreduce.job.end-notification.max.retry.interval;  =
Ignoring.
2015-02-19 19:56:08,798 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.client.conf;  =
Ignoring.
2015-02-19 19:56:08,798 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.keystores.factory.class;  =
Ignoring.
2015-02-19 19:56:08,799 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: hadoop.ssl.server.conf;  =
Ignoring.
2015-02-19 19:56:08,809 WARN [main] =
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override =
final parameter: mapreduce.job.end-notification.max.attempts;  =
Ignoring.
2015-02-19 19:56:08,821 INFO [main] =
org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at =
quickstart.cloudera/192.168.2.185:8030
2015-02-19 19:56:08,975 WARN [main] =
org.apache.hadoop.security.UserGroupInformation: =
PriviledgedActionException as:cloudera (auth:SIMPLE) =
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.to=
ken.SecretManager$InvalidToken): appattempt_1424003606313_0012_000002 =
not found in AMRMTokenSecretManager.
2015-02-19 19:56:08,976 WARN [main] =
org.apache.hadoop.ipc.Client: Exception encountered while connecting to =
the server : =
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Se=
cretManager$InvalidToken): appattempt_1424003606313_0012_000002 not =
found in AMRMTokenSecretManager.
2015-02-19 19:56:08,976 WARN [main] =
org.apache.hadoop.security.UserGroupInformation: =
PriviledgedActionException as:cloudera (auth:SIMPLE) =
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.to=
ken.SecretManager$InvalidToken): appattempt_1424003606313_0012_000002 =
not found in AMRMTokenSecretManager.
2015-02-19 19:56:08,981 ERROR [main] =
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception =
while registering
org.apache.hadoop.security.token.SecretManager$Inva=
lidToken: appattempt_1424003606313_0012_000002 not found in =
AMRMTokenSecretManager.
        at =
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native =
Method)
        at =
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc=
cessorImpl.java:57)
        at =
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst=
ructorAccessorImpl.java:45)
        at =
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at =
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)<=
o:p>
        at =
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:1=
04)
        at =
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie=
ntImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.ja=
va:109)
        at =
sun.reflect.NativeMethodAccessorImpl.invoke0(Native =
Method)
        at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:57)
        at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:43)
        at =
java.lang.reflect.Method.invoke(Method.java:606)
<=
pre>        at =
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc=
ationHandler.java:187)
        at =
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH=
andler.java:102)
        at =
com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown =
Source)
        at =
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunica=
tor.java:161)
        at =
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommu=
nicator.java:122)
        at =
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(R=
MContainerAllocator.java:238)
        at =
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)=

        at =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.s=
erviceStart(MRAppMaster.java:807)
        at =
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)=

        at =
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.=
java:120)
        at =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.j=
ava:1075)
        at =
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)=

        at =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:147=
8)
        at =
java.security.AccessController.doPrivileged(Native =
Method)
        at =
javax.security.auth.Subject.doAs(Subject.java:415)
        at =
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation=
.java:1642)
        at =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAp=
pMaster.java:1474)
        at =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407=
)
Caused by: =
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Se=
cretManager$InvalidToken): appattempt_1424003606313_0012_000002 not =
found in AMRMTokenSecretManager.
        at =
org.apache.hadoop.ipc.Client.call(Client.java:1411)
        at =
org.apache.hadoop.ipc.Client.call(Client.java:1364)
        at =
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.=
java:206)
        at =
com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown =
Source)
        at =
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie=
ntImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.ja=
va:106)
        ... 22 =
more
2015-02-19 =
19:56:08,983 INFO [main] org.apache.hadoop.service.AbstractService: =
Service RMCommunicator failed in state STARTED; cause: =
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: =
org.apache.hadoop.security.token.SecretManager$InvalidToken: =
appattempt_1424003606313_0012_000002 not found in =
AMRMTokenSecretManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeExcept=
ion: org.apache.hadoop.security.token.SecretManager$InvalidToken: =
appattempt_1424003606313_0012_000002 not found in =
AMRMTokenSecretManager.
 
 
 
 
From: Ulul [mailto:hadoop@ulul.org] =

Sent: Thursday, February 19, 2015 5:08 PM
To: user@hadoop.apache.org
S=
ubject: Re: Yarn AM is abending job when submitting a remote job to =
cluster
 
Is your point is that using the hdfs:// prefix is valid =
since our hdfs client works ?
fs.defaultFS defines the namenode =
address and the filesystem type. It doen't imply that the prefix should =
be used for yarn and mapreduce options that are not directly linked to =
hdfs 


Le =
19/02/2015 22:56, Ulul a =E9crit :
In that case it's just between your hdfs client, the NN =
and the DNs, no YARN or MR component involved.
The fact that this =
works is not related to your MR job not =
succeeding.


Le 19/02/2015 22:45, roland.depratti a =
=E9crit :
Thanks for looking at my =
problem.
 
I =
can run an hdfs command from the client, with the config file listed, =
that does a cat on a file in hdfs on the remote cluster and returns the =
contents of that file to the client.
 
- =
rd
 
 
Sent =
from my Verizon Wireless 4G LTE =
smartphone


-------- Original message --------
From: =
Ulul <hadoop@ulul.org> =

Date:02/19/2015 4:03 PM (GMT-05:00) 
To: user@hadoop.apache.org =

Subject: Re: Yarn AM is abending job when submitting a remote job to =
cluster 

Hi
Doesn't seem like an ssl error to me (the log =
states that attempts to 
override final properties are =
ignored)

On the other hand the configuration seems wrong =

:mapreduce.jobtracker.address and yarn.resourcemanager.address =
should 
only contain an IP or a hostname. You should remove 'hdfs://' =
though the 
log doesn't suggest it has anything to do with your =
problem....

And what do you mean by an "HDFS job" =
?

Ulul

Le 19/02/2015 04:22, daemeon reiydelle a =E9crit =
:
> I would guess you do not have your ssl certs set up, client or =
server, 
> based on the error.
>
> ***
> =
.......
> ***“Life should not be a journey to the grave with =
the intention of 
> arriving safely in a
> pretty and well =
preserved body, but rather to skid in broadside in a 
> cloud of =
smoke,
> thoroughly used up, totally worn out, and loudly =
proclaiming “Wow! 
> What a Ride!”*
> - Hunter =
Thompson
>
> Daemeon C.M. Reiydelle
> USA (+1) =
415.501.0198
> London (+44) (0) 20 8144 9872*/
> =
/
>
> On Wed, Feb 18, 2015 at 5:19 PM, Roland DePratti =

> <roland.depratti@cox.net <mailto:roland.depratti@cox.ne=
t>> wrote:
>
>     I have been =
searching for a handle on a problem without =
very
>     little clues. Any help pointing me =
to the right direction will be
>     =
huge.
>
>     I have not received any =
input form the Cloudera google groups.
>     =
Perhaps this is more Yarn based and I am hoping I have more luck =
here.
>
>     Any help is greatly =
appreciated.
>
>     I am running a =
Hadoop cluster using CDH5.3. I also have a =
client
>     machine with a standalone one =
node setup (VM).
>
>     All =
environments are running CentOS =
6.6.
>
>     I have submitted some Java =
mapreduce jobs locally on both the
>     =
cluster and the standalone environment with successfully =
completions.
>
>     I can submit a =
remote HDFS job from client to cluster using =
-conf
>     hadoop-cluster.xml (see below) and =
get data back from the cluster
>     with no =
problem.
>
>     When submitted remotely =
the mapreduce jobs remotely, I get an AM
>     =
error:
>
>     AM fails the job with the =
error:
>
>
>       &=
nbsp;        =
SecretManager$InvalidToken:
>     =
appattempt_1424003606313_0001_000002 not found =
in
>     =
AMRMTokenSecretManager
>
>
>     I =
searched /var/log/secure on the client and cluster with =
no
>     unusual =
messages.
>
>     Here is the contents =
of hadoop-cluster.xml:
>
>     <?xml =
version=3D"1.0" =
encoding=3D"UTF-8"?>
>
>    =
 <!--generated by Roland-->
>     =
<configuration>
>       =
<property>
>         =
<name>fs.defaultFS</name>
>    &nbs=
p;    =
<value>hdfs://mycluser:8020</value>
>   =
    =
</property>
>       =
<property>
>     =
<name>mapreduce.jobtracker.address</name>
>  =
       =
<value>hdfs://mycluster:8032</value>
>   =
;    =
</property>
>       =
<property>
>     =
<name>yarn.resourcemanager.address</name>
>  =
       =
<value>hdfs://mycluster:8032</value>
>   =
;    =
</property>
>
>     Here is the =
output from the job log on the =
cluster:
>
>     2015-02-15 07:51:06,544 =
INFO [main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: =
Created
>     MRAppMaster for application =
appattempt_1424003606313_0001_000002
>
>   &nb=
sp; 2015-02-15 07:51:06,949 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.require.client.cert;  =
Ignoring.
>
>     2015-02-15 =
07:51:06,952 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final =
parameter:
>     =
mapreduce.job.end-notification.max.retry.interval; =
Ignoring.
>
>     2015-02-15 =
07:51:06,952 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.client.conf;  =
Ignoring.
>
>     2015-02-15 =
07:51:06,954 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.keystores.factory.class; 
>     =
Ignoring.
>
>     2015-02-15 =
07:51:06,957 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.server.conf;  =
Ignoring.
>
>     2015-02-15 =
07:51:06,973 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final =
parameter:
>     =
mapreduce.job.end-notification.max.attempts; =
Ignoring.
>
>     2015-02-15 =
07:51:07,241 INFO [main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with =
tokens:
>
>     2015-02-15 07:51:07,241 =
INFO [main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: =
Kind:
>     YARN_AM_RM_TOKEN, Service: , =
Ident:
>     (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33be1aa0)
=
>
>     2015-02-15 07:51:07,332 INFO =
[main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using =
mapred
>     =
newApiCommitter.
>
>     2015-02-15 =
07:51:07,627 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.require.client.cert;  =
Ignoring.
>
>     2015-02-15 =
07:51:07,632 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final =
parameter:
>     =
mapreduce.job.end-notification.max.retry.interval; =
Ignoring.
>
>     2015-02-15 =
07:51:07,632 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.client.conf;  =
Ignoring.
>
>     2015-02-15 =
07:51:07,639 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.keystores.factory.class; 
>     =
Ignoring.
>
>     2015-02-15 =
07:51:07,645 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final parameter: =
hadoop.ssl.server.conf;  =
Ignoring.
>
>     2015-02-15 =
07:51:07,663 WARN [main]
>     =
org.apache.hadoop.conf.Configuration: job.xml:an attempt =
to
>     override final =
parameter:
>     =
mapreduce.job.end-notification.max.attempts; =
Ignoring.
>
>     2015-02-15 =
07:51:08,237 WARN [main]
>     =
org.apache.hadoop.util.NativeCodeLoader: Unable to =
load
>     native-hadoop library for your =
platform... using builtin-java
>     classes =
where applicable
>
>     2015-02-15 =
07:51:08,429 INFO [main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: =
OutputCommitter
>     set in config =
null
>
>     2015-02-15 07:51:08,499 =
INFO [main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter =
is
>     =
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
>
>=
;     2015-02-15 07:51:08,526 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.jobhistory.EventType for =
class
>     =
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
>
=
>     2015-02-15 07:51:08,527 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType =
for
>     =
class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
>=

>     2015-02-15 07:51:08,561 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType =
for
>     =
class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
>=
;
>     2015-02-15 07:51:08,562 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType
>=
     for class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher=

>
>     2015-02-15 07:51:08,566 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType =
for
>     class =
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
>>     2015-02-15 07:51:08,568 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType
>=
     for class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher<=
br>>
>     2015-02-15 07:51:08,568 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType
>=
;     for class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter>
>     2015-02-15 07:51:08,570 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType>     for class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter>
>     2015-02-15 07:51:08,599 INFO =
[main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery =
is
>     enabled. Will try to recover from =
previous life on best effort =
basis.
>
>     2015-02-15 07:51:08,642 =
INFO [main]
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous =
history
>     file is =
at
>     =
hdfs://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/j=
ob_1424003606313_0001_1.jhist
>     <http://mycluster.com=
:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_000=
1_1.jhist2015-02-15>
>
>     =
_2015-02-15
>     <http://mycluster.com=
:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_000=
1_1.jhist2015-02-15>_07:51:09,147
>     =
INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: =
Read
>     completed tasks from history =
0
>
>     2015-02-15 07:51:09,193 INFO =
[main]
>     =
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering =
class
>     =
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type
>&=
nbsp;    for class
>     =
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
&=
gt;
>     2015-02-15 07:51:09,222 INFO =
[main]
>     =
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded =
properties
>     from =
hadoop-metrics2.properties
>
>     =
2015-02-15 07:51:09,277 INFO [main]
 
 
------=_NextPart_000_00D8_01D04C88.4F64E560--