hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
Date Mon, 30 Sep 2013 22:44:24 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782364#comment-13782364
] 

Robert Kanter commented on MAPREDUCE-5088:
------------------------------------------

We're seeing an issue where Oozie can't submit to a secure cluster when the JHS is down because
it can't get a delegation token from the Job History Server.  However, if I submit a job without
Oozie (i.e. hadoop jar ....) it works fine.  When investigating, it looks like this JIRA may
be the cause, or is at least related.  Any ideas on why this is happening?

{noformat}
2013-09-30 13:14:48,544  WARN ActionStartXCommand:542 - USER[testuser] GROUP[-] TOKEN[] APP[map-reduce-wf]
JOB[0000055-130929201011702-oozie-oozi-W] ACTION[0000055-130929201011702-oozie-oozi-W@mr-node]
Er
ror starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [  JA006], Message [  JA006:
Connection refused]
org.apache.oozie.action.ActionExecutorException:   JA006: Connection refused
        at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
        at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
        at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:794)
        at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:948)
        at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:215)
        at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60)
        at org.apache.oozie.command.XCommand.call(XCommand.java:280)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
        at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
        at org.apache.hadoop.ipc.Client.call(Client.java:1318)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy31.getDelegationToken(Unknown Source)
        at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
        at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:208)
        at org.apache.hadoop.mapred.YARNRunner.addHistoryToken(YARNRunner.java:195)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:280)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:552)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:552)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:543)
        at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:779)
        ... 10 more
{noformat}

> MR Client gets an renewer token exception while Oozie is submitting a job
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5088
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.0.3-alpha
>            Reporter: Roman Shaposhnik
>            Assignee: Daryn Sharp
>            Priority: Blocker
>             Fix For: 2.0.4-alpha
>
>         Attachments: HADOOP-9409.patch, HADOOP-9409.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.patch,
MAPREDUCE-5088.txt
>
>
> After the fix for HADOOP-9299 I'm now getting the following bizzare exception in Oozie
while trying to submit a job. This also seems to be KRB related:
> {noformat}
> 2013-03-15 13:34:16,555  WARN ActionStartXCommand:542 - USER[hue] GROUP[-] TOKEN[] APP[MapReduce]
JOB[0000001-130315123130987-oozie-oozi-W] ACTION[0000001-130315123130987-oozie-oozi-W@Sleep]
Error starting action [Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
Message [UninitializedMessageException: Message missing required fields: renewer]
> org.apache.oozie.action.ActionExecutorException: UninitializedMessageException: Message
missing required fields: renewer
> 	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
> 	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
> 	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
> 	at org.apache.oozie.command.XCommand.call(XCommand.java:277)
> 	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
> 	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
> 	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.UninitializedMessageException: Message missing required
fields: renewer
> 	at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
> 	at org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
> 	at org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
> 	at org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
> 	at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
> 	at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
> 	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> 	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
> 	... 10 more
> 2013-03-15 13:34:16,555  WARN ActionStartXCommand:542 - USER[hue] GROUP[-] TOKEN[] APP[MapReduce]
JOB[0000001-13031512313
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message