hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
Date Fri, 05 Aug 2011 15:26:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080014#comment-13080014

Daryn Sharp commented on MAPREDUCE-2764:

bq. The token is used for rpc connection and when token selector does a match it needs the
service to have rpc port. But if we store hftp uri in the token, the port will be http port.
How is that being handled?

The {{getServiceAuthority()}} is extracting the authority from the service if it's a uri,
else it uses the service itself if it's not a uri.

bq. One approach could be to store both issuer and service in the current service field.

The the service is currently "authority", and I changed it to be either "authority" or "scheme://authority".
 I'd like to eventually move everything to "scheme://authority" hence the compatibility.

bq. I think the AbstractDelegationtionTokenSelector change and introduction of TokenMatcher
is unnecessary.  BlockTokenSelector or JobTokenSelector shouldn't need to change at all.

That means the token selectors go back to copy-n-paste...  The base {{selectTokens()}} method
filters the tokens to find the expected token kind.  This eliminates the copy-n-paste code
which limits the ability to alter the standard behavior.

The {{TokenMatcher}} is used to allow tokens, if those so chose, to provide custom selection
criteria above and beyond the standard service match.  This is also used to match a RPC socket
to its token, else it just adds even more copy-n-paste.

{{JobTokenSelector}} only removed methods now in the base class.  {{BlockTokenSelector}} oddly
doesn't care about service at all, so it's {{selectToken()}} is overridden to provide a {{TokenMatcher}}
that ignores service.

bq. A change in hdfs's DelegationTokenSelector should suffice for this jira.

Iff we are going back to copy-n-paste...

> Fix renewal of dfs delegation tokens
> ------------------------------------
>                 Key: MAPREDUCE-2764
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For:
>         Attachments: MAPREDUCE-2764.patch
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  The problem
is the JT's delegation token renewal code is built on brittle assumptions.  The token's service
field contains only the "ip:port" pair.  The renewal process assumes that the scheme must
be hdfs.  If that fails due to a {{VersionMismatchException}}, it tries https based on another
assumption that it must be hftp if it's not hdfs.  A number of other exceptions, most commonly
{{IOExceptions}}, can be generated which fouls up the renewal since it won't fallback to https.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message