hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sunmeng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11976) submit job with oozie to between cluster
Date Fri, 15 May 2015 10:07:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545262#comment-14545262
] 

sunmeng commented on HADOOP-11976:
----------------------------------


Logged in as: dr.who 
About Apache Hadoop
Application
About
Jobs
Tools
Log Type: syslog
Log Length: 29472
2015-05-14 14:45:41,956 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created
MRAppMaster for application appattempt_1431573307666_0426_000001
2015-05-14 14:45:42,498 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.require.client.cert;  Ignoring.
2015-05-14 14:45:42,503 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-05-14 14:45:42,505 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.client.conf;  Ignoring.
2015-05-14 14:45:42,508 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.keystores.factory.class;  Ignoring.
2015-05-14 14:45:42,515 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.server.conf;  Ignoring.
2015-05-14 14:45:42,544 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-05-14 14:45:42,707 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing
with tokens:
2015-05-14 14:45:42,707 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@344d638a)
2015-05-14 14:45:42,748 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind:
RM_DELEGATION_TOKEN, Service: 10.10.11.248:8032, Ident: (owner=mengsun, renewer=oozie mr token,
realUser=oozie, issueDate=1431585938658, maxDate=1432190738658, sequenceNumber=2, masterKeyId=2)
2015-05-14 14:45:43,467 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.require.client.cert;  Ignoring.
2015-05-14 14:45:43,470 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-05-14 14:45:43,471 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.client.conf;  Ignoring.
2015-05-14 14:45:43,472 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.keystores.factory.class;  Ignoring.
2015-05-14 14:45:43,475 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.server.conf;  Ignoring.
2015-05-14 14:45:43,488 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-05-14 14:45:44,093 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where applicable
2015-05-14 14:45:44,316 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter
set in config null
2015-05-14 14:45:44,318 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter
is org.apache.hadoop.mapred.FileOutputCommitter
2015-05-14 14:45:44,387 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2015-05-14 14:45:44,389 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2015-05-14 14:45:44,390 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2015-05-14 14:45:44,391 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2015-05-14 14:45:44,392 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2015-05-14 14:45:44,393 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2015-05-14 14:45:44,394 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-05-14 14:45:44,396 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2015-05-14 14:45:44,436 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Perms after creating 488, Expected: 504
2015-05-14 14:45:44,436 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Explicitly setting permissions to : 504, rwxrwx---
2015-05-14 14:45:44,508 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering
class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2015-05-14 14:45:44,802 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties
2015-05-14 14:45:44,888 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2015-05-14 14:45:44,888 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster
metrics system started
2015-05-14 14:45:44,907 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding
job token for job_1431573307666_0426 to jobTokenSecretManager
2015-05-14 14:45:45,639 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not
uberizing job_1431573307666_0426 because: not enabled;
2015-05-14 14:45:45,669 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input
size for job job_1431573307666_0426 = 0. Number of splits = 1
2015-05-14 14:45:45,669 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number
of reduces for job job_1431573307666_0426 = 0
2015-05-14 14:45:45,669 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1431573307666_0426Job
Transitioned from NEW to INITED
2015-05-14 14:45:45,671 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster
launching normal, non-uberized, multi-container job job_1431573307666_0426.
2015-05-14 14:45:45,707 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue
class java.util.concurrent.LinkedBlockingQueue
2015-05-14 14:45:45,717 INFO [Socket Reader #1 for port 59500] org.apache.hadoop.ipc.Server:
Starting Socket Reader #1 for port 59500
2015-05-14 14:45:45,743 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl:
Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2015-05-14 14:45:45,743 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-05-14 14:45:45,743 INFO [IPC Server listener on 59500] org.apache.hadoop.ipc.Server:
IPC Server listener on 59500: starting
2015-05-14 14:45:45,744 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService:
Instantiated MRClientService at bja-pro0070.hadoop.cpcc.iflyyun.cn/10.10.11.20:59500
2015-05-14 14:45:45,824 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
2015-05-14 14:45:45,829 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log
for http.requests.mapreduce is not defined
2015-05-14 14:45:45,841 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter
'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-05-14 14:45:45,849 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2015-05-14 14:45:45,850 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2015-05-14 14:45:45,854 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec:
/mapreduce/*
2015-05-14 14:45:45,855 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec:
/ws/*
2015-05-14 14:45:45,866 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port
60571
2015-05-14 14:45:45,866 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2015-05-14 14:45:45,896 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/hadoop-yarn-common-2.5.0-cdh5.3.2.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_60571_mapreduce____.3n0f7g/webapp
2015-05-14 14:45:46,309 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:60571
2015-05-14 14:45:46,310 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce
started at 60571
2015-05-14 14:45:46,704 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp
guice modules
2015-05-14 14:45:46,709 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue
class java.util.concurrent.LinkedBlockingQueue
2015-05-14 14:45:46,710 INFO [Socket Reader #1 for port 52266] org.apache.hadoop.ipc.Server:
Starting Socket Reader #1 for port 52266
2015-05-14 14:45:46,718 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-05-14 14:45:46,718 INFO [IPC Server listener on 52266] org.apache.hadoop.ipc.Server:
IPC Server listener on 52266: starting
2015-05-14 14:45:46,738 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
2015-05-14 14:45:46,738 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3
2015-05-14 14:45:46,738 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33
2015-05-14 14:45:46,882 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.require.client.cert;  Ignoring.
2015-05-14 14:45:46,884 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-05-14 14:45:46,885 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.client.conf;  Ignoring.
2015-05-14 14:45:46,886 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.keystores.factory.class;  Ignoring.
2015-05-14 14:45:46,889 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: hadoop.ssl.server.conf;  Ignoring.
2015-05-14 14:45:46,902 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-05-14 14:45:46,908 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager
at hfa-pro0002.hadoop.cpcc.iflyyun.cn/10.10.12.248:8030
2015-05-14 14:45:47,036 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:mengsun (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
2015-05-14 14:45:47,037 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while
connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
2015-05-14 14:45:47,037 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:mengsun (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
2015-05-14 14:45:47,042 ERROR [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Exception while registering
org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1431573307666_0426_000001
not found in AMRMTokenSecretManager.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:238)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:807)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1075)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
	... 22 more
2015-05-14 14:45:47,046 INFO [main] org.apache.hadoop.service.AbstractService: Service RMCommunicator
failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:238)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:807)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1075)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1431573307666_0426_000001
not found in AMRMTokenSecretManager.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)
	... 14 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
	... 22 more
2015-05-14 14:45:47,049 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0
CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2015-05-14 14:45:47,049 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:238)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:807)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1075)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1431573307666_0426_000001
not found in AMRMTokenSecretManager.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)
	... 14 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
	... 22 more
2015-05-14 14:45:47,051 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster
failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:238)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:807)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1075)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1431573307666_0426_000001
not found in AMRMTokenSecretManager.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
	at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)
	... 14 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1431573307666_0426_000001 not found in AMRMTokenSecretManager.
	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
	... 22 more
2015-05-14 14:45:47,053 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Stopping JobHistoryEventHandler. Size of the outstanding queue size is 2
2015-05-14 14:45:47,053 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event AM_STARTED
2015-05-14 14:45:47,179 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Event Writer setup for JobId: job_1431573307666_0426, File: hdfs://bja-pro0002.hadoop.cpcc.iflyyun.cn:8020/user/mengsun/.staging/job_1431573307666_0426/job_1431573307666_0426_1.jhist
2015-05-14 14:45:47,784 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
In stop, writing event JOB_SUBMITTED
2015-05-14 14:45:47,856 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Stopped JobHistoryEventHandler. super.stop()
2015-05-14 14:45:47,856 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Skipping
cleaning up the staging dir. assuming AM will be retried.
2015-05-14 14:45:47,856 INFO [main] org.apache.hadoop.ipc.Server: Stopping server on 52266
2015-05-14 14:45:47,858 INFO [IPC Server listener on 52266] org.apache.hadoop.ipc.Server:
Stopping IPC Server listener on 52266
2015-05-14 14:45:47,859 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler:
TaskHeartbeatHandler thread interrupted
2015-05-14 14:45:47,859 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping
IPC Server Responder


> submit job with oozie to between cluster
> ----------------------------------------
>
>                 Key: HADOOP-11976
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11976
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: sunmeng
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message