hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
Date Wed, 31 Aug 2011 19:10:11 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094802#comment-13094802

Owen O'Malley commented on MAPREDUCE-2764:

The service field was a per-kind specific information that contained information about which
instance of the service generated the token. It is used in two cases:
  1. picking which token to use when opening an rpc connection to the service (eg. NameNode,
JobTracker, Oozie, HBase, ...)
  2. connecting to the service to renew or cancel the token

For HDFS delegation tokens the service consists of the host and port of the NameNode. For
HFTP delegation tokens, it would have been ideal if it had always contained the host and port
of the HFTP server. Then all of the mapping would have been done in the HFTP servlet code
as it converted an HDFS token into a HFTP one. Since the access mechanism is *totally* different
for the two types of tokens, they should be pulled apart into separate kinds so that they
can have different renewal/canceling mechanisms.

Given that 203 and 204 are in production, we can't make incompatible changes, therefore the
HFTP service needs to continue to send and receive HDFS delegation tokens. Therefore, having
the HFTP client manipulate the tokens into the form that *should* have been sent seems like
the best alternative.

> Fix renewal of dfs delegation tokens
> ------------------------------------
>                 Key: MAPREDUCE-2764
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For:
>         Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, delegation.patch
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  The problem
is the JT's delegation token renewal code is built on brittle assumptions.  The token's service
field contains only the "ip:port" pair.  The renewal process assumes that the scheme must
be hdfs.  If that fails due to a {{VersionMismatchException}}, it tries https based on another
assumption that it must be hftp if it's not hdfs.  A number of other exceptions, most commonly
{{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message