hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3825) Need generalized multi-token filesystem support
Date Mon, 13 Feb 2012 16:48:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206976#comment-13206976

Daryn Sharp commented on MAPREDUCE-3825:

By not backwards-compatible, I'm referring to the fact that external filesystems, that may
currently work fine, will need to be changed.  They must also be changed in a precise manner,
involving copy-n-paste, to not introduce bugs.

I believe the history of the api is {{getDelegationToken}} (singular) was deprecated after
{{ViewFileSystem}} broke the paradigm of 1 filesystem, 1 token.  I think we agree that the
replacement {{getDelegationTokens}}, as currently implemented in 23, is flawed.  However I
think it is on the track with a few modification.

The use cases I seek to satisfy:
# I want this filesystem's delegation token - {{getDelegationToken(renewer)}}
# I want all delegation tokens used by this filesystem - {{getDelegationTokens(renewer)}}
# I want the delegation tokens that I don't already have for this filesystem.  I want to know
which tokens were acquired - {{getDelegationTokens(renewer, creds)}}
# I want to know all filesystems used by this filesystem - {{getFileSystems()}}
# If I need to change the semantics of overall token acquisition, +I do not want to change
any filesystem+.

The counter-proposal provides:
# I want all the delegation tokens that I don't already have for this filesystem, but don't
tell me what they were - {{addDelegationTokens(renewer, creds)}}
# If I want to change the semantics of overall token acquisition, +I want to require changes
in all filesystems+.

The final list items illustrate the core difference in the approaches: Do we want a decoupled
design?  Or do we want a tightly coupled design?  We can achieve a decoupled design via use
of filesystem primitives.  This removes the onus from all filesystems to include copy-n-paste
# Obtaining a token _is_ the responsibility of a filesystem.
#* If a filesystem has a token, it should only override the +primitive+ {{getDelegationToken(renewer)}},
as it does today.
#* If a filesystem provides multiple tokens, then it should override the +primitive+ {{getFileSystems()}}.
 The default implementation is sufficient for existing filesystems, sans {{ViewFileSystem}}
which should return its mounted filesystems.
# Overall token acquisition _is not_ the responsibility of a filesystem.
#* A consistent implementation should be provided via {{final}} methods {{getDelegationTokens(renewer)}}
and {{getDelegationTokens(renewer, creds)}}

> Need generalized multi-token filesystem support
> -----------------------------------------------
>                 Key: MAPREDUCE-3825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.1, 0.24.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: MAPREDUCE-3825.patch, TokenCache.pdf
> This is the counterpart to HADOOP-7967.  The token cache currently tries to assume a
filesystem's token service key.  The assumption generally worked while there was a one to
one mapping of filesystem to token.  With the advent of multi-token filesystems like viewfs,
the token cache will try to use a service key (ie. for viewfs) that will never exist (because
it really gets the mounted fs tokens).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message