hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5262) AM generates NPEs when RM connection fails
Date Tue, 21 May 2013 16:37:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663096#comment-13663096
] 

Daryn Sharp commented on MAPREDUCE-5262:
----------------------------------------

Also, the web UI reports the job starting at the epoch:
{noformat}
Started:	 Wed Dec 31 23:59:59 UTC 1969
Finished:	 Tue May 21 00:32:03 UTC 2013
Elapsed:	 380304hrs, 32mins, 3sec
{noformat}
                
> AM generates NPEs when RM connection fails
> ------------------------------------------
>
>                 Key: MAPREDUCE-5262
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5262
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 3.0.0, 2.0.4-alpha
>            Reporter: Daryn Sharp
>
> If the AM fails to connect to the RM, it causes a cascade of NPEs as the AM attempts
to shutdown and exit.
> {noformat}
> 2013-05-21 00:31:56,153 ERROR [main] org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoopqa (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1367605529307_0034_000001
> 2013-05-21 00:31:56,154 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered
while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1367605529307_0034_000001
> 2013-05-21 00:31:56,154 ERROR [main] org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoopqa (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1367605529307_0034_000001
> 2013-05-21 00:31:56,156 ERROR [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Exception while registering
> java.lang.reflect.UndeclaredThrowableException
> 	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
> 	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:103)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:153)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.start(RMCommunicator.java:112)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.start(RMContainerAllocator.java:211)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.start(MRAppMaster.java:797)
> 	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1014)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1374)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1370)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1318)
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1367605529307_0034_000001
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1266)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1218)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> 	at com.sun.proxy.$Proxy28.registerApplicationMaster(Unknown Source)
> 	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:100)
> 	... 12 more
> 2013-05-21 00:31:56,158 ERROR [main] org.apache.hadoop.yarn.service.CompositeService:
Error starting services org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> org.apache.hadoop.yarn.YarnException: java.lang.reflect.UndeclaredThrowableException
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:166)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.start(RMCommunicator.java:112)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.start(RMContainerAllocator.java:211)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.start(MRAppMaster.java:797)
> 	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1014)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1374)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1370)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1318)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> 	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
> 	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:103)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:153)
> 	... 11 more
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1367605529307_0034_000001
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1266)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1218)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> 	at com.sun.proxy.$Proxy28.registerApplicationMaster(Unknown Source)
> 	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:100)
> 	... 12 more
> 2013-05-21 00:31:56,158 INFO [main] org.apache.hadoop.yarn.service.CompositeService:
Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
> java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.stop(RMCommunicator.java:219)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.stop(RMContainerAllocator.java:251)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.stop(MRAppMaster.java:803)
> 	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
> 	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:77)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1014)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1374)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1370)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1318)
> 2013-05-21 00:31:56,158 INFO [main] org.apache.hadoop.ipc.Server: Stopping server on
39121
> 2013-05-21 00:31:56,160 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:TaskHeartbeatHandler
is stopped.
> 2013-05-21 00:31:56,160 INFO [IPC Server listener on 39121] org.apache.hadoop.ipc.Server:
Stopping IPC Server listener on 39121
> 2013-05-21 00:31:56,160 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping
IPC Server Responder
> 2013-05-21 00:31:56,160 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler:
TaskHeartbeatHandler thread interrupted
> 2013-05-21 00:31:56,160 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapred.TaskAttemptListenerImpl
is stopped.
> 2013-05-21 00:31:56,161 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:CommitterEventHandler
is stopped.
> 2013-05-21 00:31:56,161 INFO [main] org.apache.hadoop.ipc.Server: Stopping server on
50500
> 2013-05-21 00:31:56,161 INFO [IPC Server listener on 50500] org.apache.hadoop.ipc.Server:
Stopping IPC Server listener on 50500
> 2013-05-21 00:31:56,161 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping
IPC Server Responder
> 2013-05-21 00:31:56,164 INFO [main] org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:0
> 2013-05-21 00:31:56,264 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:MRClientService
is stopped.
> 2013-05-21 00:31:56,264 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher
is stopped.
> 2013-05-21 00:31:56,264 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Error starting MRAppMaster
> org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> 	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1014)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1374)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1370)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1318)
> Caused by: org.apache.hadoop.yarn.YarnException: java.lang.reflect.UndeclaredThrowableException
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:166)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.start(RMCommunicator.java:112)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.start(RMContainerAllocator.java:211)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.start(MRAppMaster.java:797)
> 	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> 	... 7 more
> Caused by: java.lang.reflect.UndeclaredThrowableException
> 	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
> 	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:103)
> 	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:153)
> 	... 11 more
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1367605529307_0034_000001
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1266)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1218)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> 	at com.sun.proxy.$Proxy28.registerApplicationMaster(Unknown Source)
> 	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:100)
> 	... 12 more
> 2013-05-21 00:31:56,266 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.
> 2013-05-21 00:31:56,266 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
RMCommunicator notified that iSignalled is: true
> 2013-05-21 00:31:56,266 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Notify RMCommunicator isAMLastRetry: false
> 2013-05-21 00:31:56,266 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
RMCommunicator notified that shouldUnregistered is: false
> 2013-05-21 00:31:56,267 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Notify JHEH isAMLastRetry: false
> 2013-05-21 00:31:56,267 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is false
> 2013-05-21 00:31:56,267 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Stopping JobHistoryEventHandler. Size of the outstanding queue size is 3
> 2013-05-21 00:31:56,267 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event AM_STARTED
> 2013-05-21 00:31:56,347 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Event Writer setup for JobId: job_1367605529307_0035, File: hdfs://hdfs-server:8020/user/hadoopqa/.staging/job_1367605529307_0035/job_1367605529307_0035_2.jhist
> 2013-05-21 00:31:56,356 WARN [Thread-1] org.apache.hadoop.conf.Configuration: user.name
is deprecated. Instead, use mapreduce.job.user.name
> 2013-05-21 00:31:56,570 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event AM_STARTED
> 2013-05-21 00:31:56,571 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event JOB_SUBMITTED
> 2013-05-21 00:31:56,588 INFO [Thread-1] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Stopped JobHistoryEventHandler. super.stop()
> 2013-05-21 00:31:56,588 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Skipping cleaning up the staging dir. assuming AM will be retried.
> 2013-05-21 00:31:56,588 INFO [Thread-1] org.apache.hadoop.yarn.service.CompositeService:
Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
> java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter.stop(MRAppMaster.java:865)
> 	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
> 	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1343)
> 	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2013-05-21 00:31:56,588 INFO [Thread-1] org.apache.hadoop.ipc.Server: Stopping server
on 39121
> 2013-05-21 00:31:56,588 INFO [Thread-1] org.apache.hadoop.ipc.Server: Stopping server
on 50500
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message