hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9409) MR Client gets an renewer token exception while Oozie is submitting a job
Date Mon, 18 Mar 2013 13:40:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13605120#comment-13605120
] 

Daryn Sharp commented on HADOOP-9409:
-------------------------------------

If we hope to allow a client to work with heterogenous clusters with security enabled/disabled,
then clients must properly deal with a null response to a token request.  As an aside: in
a generic sense, null means "no token is required" which currently means security is disabled
but it doesn't necessarily have to mean that.  Semi-recent changes to MR will propagate credentials
(tokens & secrets), even if security is off, because 1.x did that and I think hive was
relying on it to pass a secret.  So a heterogeneous security mix will "sorta kinda" work.
 Job submission really shouldn't be shorting out the request for fs tokens because it already
properly handles null responses.

Given the error, the job submission explicitly requested a RM token which "succeeded", which
triggered an internal fetch of a HS token.  The implicit HS token request's renewer is the
value of {{yarn.resourcemanager.principal}} which in this case appears to be null.  A RM token
request requires an explicit renewer which should be that exact same value - which implies
that oozie is using a different value or the RM protobuf is allowing a null (valid).

So we need to know what conf value oozie is passing as the renewer for a RM token.  Technically
I think oozie or other components shouldn't have to supply the renewer, the job submission
should use the right value but that's a separate issue.  So for now, this may be an oozie
issue and/or due to {{yarn.resourcemanager.principal}} not being set.
                
> MR Client gets an renewer token exception while Oozie is submitting a job
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-9409
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9409
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.0.3-alpha
>            Reporter: Roman Shaposhnik
>            Priority: Blocker
>             Fix For: 2.0.4-alpha
>
>
> After the fix for HADOOP-9299 I'm now getting the following bizzare exception in Oozie
while trying to submit a job. This also seems to be KRB related:
> {noformat}
> 2013-03-15 13:34:16,555  WARN ActionStartXCommand:542 - USER[hue] GROUP[-] TOKEN[] APP[MapReduce]
JOB[0000001-130315123130987-oozie-oozi-W] ACTION[0000001-130315123130987-oozie-oozi-W@Sleep]
Error starting action [Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
Message [UninitializedMessageException: Message missing required fields: renewer]
> org.apache.oozie.action.ActionExecutorException: UninitializedMessageException: Message
missing required fields: renewer
> 	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
> 	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
> 	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
> 	at org.apache.oozie.command.XCommand.call(XCommand.java:277)
> 	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
> 	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
> 	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.UninitializedMessageException: Message missing required
fields: renewer
> 	at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
> 	at org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
> 	at org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
> 	at org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
> 	at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
> 	at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
> 	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> 	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
> 	... 10 more
> 2013-03-15 13:34:16,555  WARN ActionStartXCommand:542 - USER[hue] GROUP[-] TOKEN[] APP[MapReduce]
JOB[0000001-13031512313
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message