hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3825) MR should not be getting duplicate tokens for a MR Job.
Date Mon, 13 Feb 2012 18:05:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207019#comment-13207019
] 

Daryn Sharp commented on MAPREDUCE-3825:
----------------------------------------

bq. Good point but the new apis in 23 or the new APIs you suggested will not be implemented
by the external file systems and there is no default impl that works except for "not implemented"
exception.

Yes, my proposal will work because existing filesystems don't need to change.
* A filesystem need only implement {{getDelegationToken}} as it does today.  Backwards-compatible
* A filesystem optionally implements {{getFileSystems}}.  The default returns the filesystem
itself.  Backwards-compatible.  Only {{ViewFileSystem}} needs to override {{getFileSystems}}.
* {{getDelegationTokens}} should be common code that no filesystem needs to override, which
is why I'm suggesting it as a {{final}} method.
                
> MR should not be getting duplicate tokens for a MR Job.
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-3825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.1, 0.24.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: MAPREDUCE-3825.patch, TokenCache.pdf
>
>
> This is the counterpart to HADOOP-7967.  
> MR gets tokens for all input, output and the default filesystem when a MR job is submitted.

> The APIs in FileSystem make it challenging to avoid duplicate tokens when there are file
systems that have embedded
> filesystems.
> Here is the original description that Daryn wrote: 
> The token cache currently tries to assume a filesystem's token service key.  The assumption
generally worked while there was a one to one mapping of filesystem to token.  With the advent
of multi-token filesystems like viewfs, the token cache will try to use a service key (ie.
for viewfs) that will never exist (because it really gets the mounted fs tokens).
> The descriop

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message