hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
Date Fri, 05 Aug 2011 06:49:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079804#comment-13079804

Jitendra Nath Pandey commented on MAPREDUCE-2764:

   The token is used for rpc connection and when token selector does a match it needs the
service to have rpc port. But if we store hftp uri in the token, the port will be http port.
How is that being handled?
   In my opinion the basic problem is that we are using the same service field for two different
purposes. One purpose is where token should be used and other is to store how token was issued.
That is why I suggested to introduce additional issuer field. But backward compatibility issue
is a good point. One approach could be to store both issuer and service in the current service

  I think the AbstractDelegationtionTokenSelector change and introduction of TokenMatcher
is unnecessary. BlockTokenSelector or JobTokenSelector shouldn't need to change at all. A
change in hdfs's DelegationTokenSelector should suffice for this jira.

> Fix renewal of dfs delegation tokens
> ------------------------------------
>                 Key: MAPREDUCE-2764
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For:
>         Attachments: MAPREDUCE-2764.patch
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  The problem
is the JT's delegation token renewal code is built on brittle assumptions.  The token's service
field contains only the "ip:port" pair.  The renewal process assumes that the scheme must
be hdfs.  If that fails due to a {{VersionMismatchException}}, it tries https based on another
assumption that it must be hftp if it's not hdfs.  A number of other exceptions, most commonly
{{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message