hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3825) Need generalized multi-token filesystem support
Date Wed, 08 Feb 2012 15:19:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203658#comment-13203658

Daryn Sharp commented on MAPREDUCE-3825:

I should clarify how I implemented #1.  Conceptually it does exactly what is proposed.
* {{FileSystem#getDelegationTokens(renewer)}} unconditionally acquires all tokens
* {{FileSystem#getDelegationTokens(renewer, creds)}} acquires only the tokens that are missing
from the creds.  I would rather have added them to the creds and returned just the new tokens.
 The idea was previously turned down, so I was striving for compatibility with existing api.
 Note: existing impl causes token cache to log it acquired the same token multiple times when
it really hasn't.  The method should probably add new tokens to the creds, and return only
the new tokens instead of the full set.
* {{FileSystem#getDelegationTokens(renewer, creds)}} gets a unique list of leaf filesystems
and then gets their tokens
* {{FileSystems#getEmbeddedFileSystems()}} is implemented w/o the word embedded and returns
filesystems.  Returning a list of URIs isn't desirable since it forces a {{FileSystem.get}}
which may create a new fs instance if the cache is off, or creates unnecessary overhead if
the cache is on.  Both normal filesystems and viewfs already have instantiations of the filesystems
so I thought it makes sense to return them.

bq.  At end call getDelegationTokens(renewer) for each of non-dulicate URIs; getDelegationTokens
will return only one token since it is guaranteed that the calls are for the leaf file systems.

The patch does acquire tokens for only the unique set of leaf filesystems.  If {{getDelegationTokens}}
is required to return only one token, then why not call {{getDelegationToken}} which is the
existing method implemented by filesystems?  I don't think we want all subclasses to override
{{getDelegationTokens}} since it will force them to copy-n-paste the default impl with a one
line tweak.

> Need generalized multi-token filesystem support
> -----------------------------------------------
>                 Key: MAPREDUCE-3825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.1, 0.24.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
> This is the counterpart to HADOOP-7967.  The token cache currently tries to assume a
filesystem's token service key.  The assumption generally worked while there was a one to
one mapping of filesystem to token.  With the advent of multi-token filesystems like viewfs,
the token cache will try to use a service key (ie. for viewfs) that will never exist (because
it really gets the mounted fs tokens).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message