hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3825) Need generalized multi-token filesystem support
Date Mon, 13 Feb 2012 16:48:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206976#comment-13206976
] 

Daryn Sharp commented on MAPREDUCE-3825:
----------------------------------------

By not backwards-compatible, I'm referring to the fact that external filesystems, that may
currently work fine, will need to be changed.  They must also be changed in a precise manner,
involving copy-n-paste, to not introduce bugs.

I believe the history of the api is {{getDelegationToken}} (singular) was deprecated after
{{ViewFileSystem}} broke the paradigm of 1 filesystem, 1 token.  I think we agree that the
replacement {{getDelegationTokens}}, as currently implemented in 23, is flawed.  However I
think it is on the track with a few modification.

The use cases I seek to satisfy:
# I want this filesystem's delegation token - {{getDelegationToken(renewer)}}
# I want all delegation tokens used by this filesystem - {{getDelegationTokens(renewer)}}
# I want the delegation tokens that I don't already have for this filesystem.  I want to know
which tokens were acquired - {{getDelegationTokens(renewer, creds)}}
# I want to know all filesystems used by this filesystem - {{getFileSystems()}}
# If I need to change the semantics of overall token acquisition, +I do not want to change
any filesystem+.

The counter-proposal provides:
# I want all the delegation tokens that I don't already have for this filesystem, but don't
tell me what they were - {{addDelegationTokens(renewer, creds)}}
# If I want to change the semantics of overall token acquisition, +I want to require changes
in all filesystems+.

The final list items illustrate the core difference in the approaches: Do we want a decoupled
design?  Or do we want a tightly coupled design?  We can achieve a decoupled design via use
of filesystem primitives.  This removes the onus from all filesystems to include copy-n-paste
logic.
# Obtaining a token _is_ the responsibility of a filesystem.
#* If a filesystem has a token, it should only override the +primitive+ {{getDelegationToken(renewer)}},
as it does today.
#* If a filesystem provides multiple tokens, then it should override the +primitive+ {{getFileSystems()}}.
 The default implementation is sufficient for existing filesystems, sans {{ViewFileSystem}}
which should return its mounted filesystems.
# Overall token acquisition _is not_ the responsibility of a filesystem.
#* A consistent implementation should be provided via {{final}} methods {{getDelegationTokens(renewer)}}
and {{getDelegationTokens(renewer, creds)}}

Thoughts?
                
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
>                 Key: MAPREDUCE-3825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.1, 0.24.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: MAPREDUCE-3825.patch, TokenCache.pdf
>
>
> This is the counterpart to HADOOP-7967.  The token cache currently tries to assume a
filesystem's token service key.  The assumption generally worked while there was a one to
one mapping of filesystem to token.  With the advent of multi-token filesystems like viewfs,
the token cache will try to use a service key (ie. for viewfs) that will never exist (because
it really gets the mounted fs tokens).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message