hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5375) Delegation Token renewal exception in jobtracker logs
Date Thu, 04 Jul 2013 18:01:53 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700263#comment-13700263
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-5375:
----------------------------------------------------

Debugged this with [~venkatnrangan] and [~sseth]. There are a few issues here.

One is the HDFS delegation token renewal issue. Oozie does not handle the auth to locals mapping
configuration correctly. Furthermore, there can be multiple version of this auth to local
configuration in different clusters that Oozie has (specified by the oozie configuration property
oozie.service.HadoopAccessorService.hadoop.configurations). The entries are keyed by the host:port
of the jt/nn host:port but there is a default catch-all configuration if specific hadoop configuration
is not specified. The fix on oozie side is to use the oozie catch-all configuration (specified
by *= in the oozie hadoop config mapping entry) to have the right auth to local mapping and
use it in all the clusters that oozie is launching jobs.

There is MR side delegation issue to. Irrespective of the renewer principal Oozie sets, the
JobTracker changes the token to a local name but then expects it to be a full principal when
renewwing. So, obviously Oozie cannot satisfy this requirement and the MR delegation issue
has to be fixed in Jobtracker and the HDFS delegation issue will be addressed in Oozie.

                
> Delegation Token renewal exception in jobtracker logs
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-5375
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5375
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Venkat Ranganathan
>            Assignee: Venkat Ranganathan
>            Priority: Critical
>
> Filing on behalf of [~venkatnrangan] who found this originally and provided a patch.
> Saw this in the JT logs while oozie tests were running with Hadoop.
> When Oozie java action is executed, the following shows up in the job tracker log.
> {code}
> ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing
tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65 64 26 6f 6f 7a 69 65 2f 63 6f 6e
64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a
01 3e a6 87 5e 5b 8a 01 3e ca 93 e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: ip:50300.
Not rescheduled
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException:
Client jt/host@domain.com tries to renew a token with renewer specified as mapred
>         at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267)
>         at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>         at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>         at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source)
>         at org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578)
>         at org.apache.hadoop.security.token.Token.renew(Token.java:309)
>         at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221)
>         at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216)
>         at java.util.TimerThread.mainLoop(Timer.java:512)
>         at java.util.TimerThread.run(Timer.java:462)
> {code}
> Setting the renewer to Kerberos Local name does not help because AbstractDelegationTokenIdentifier
sets the renewer to Kerberos shortname but JobTracker.renewDelegationToken uses the fullName.
 This essentially causes the renewal to fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message