hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Paduano (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
Date Mon, 14 Dec 2015 23:40:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056987#comment-15056987

Matthew Paduano commented on HADOOP-12563:

I agree with the comment about addDelegationTokens.   I changed the interface in DtFetcher
use this more general signature and included the Credentials object in the interface so that
DtFetcher implementation class may have full control over how the tokens are placed into the

credentials map.

Re -fs and the abstraction used here:   I agree that ServiceLoader<DtFetcher> is "reabstracting"
one method (getDelegationToken) from the FileSystem layer.  FileSystem et al are nicely abstracted.
But FileSystem is not generalized to other services (e.g. Yarn).   I think what is needed
here is a 
generalization of that FileSystem abstraction.   getServiceName, or something like it, is
as a key to identify the implementation to use (this could be the generalized analog of -fs,
that analog exists).

Perhaps someone can think about the proper way to generalize implementations across
different projects/services so that new service specific tools can be more cleanly abstracted.
think this is a separate JIRA from this one though.  This ticket changes the serialization
of the token files and adds the ability to add multiple tokens to a single file.

The alias thing is a bit of a hack.  We have discussed adding alias as a new field (or using

the key from the credentials object) rather than overwriting the service field in the token.
in order to keep the scope of this ticket more limited, we decided to leave the behavior as
is for now so that we can get this new functionality rolled out.  Changing that behavior will
involve checking all the places Token is used.

> Updated utility to create/modify token files
> --------------------------------------------
>                 Key: HADOOP-12563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12563
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: Matthew Paduano
>         Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, HADOOP-12563.03.patch,
HADOOP-12563.04.patch, HADOOP-12563.05.patch, HADOOP-12563.06.patch, example_dtutil_commands_and_output.txt,
> hdfs fetchdt is missing some critical features and is geared almost exclusively towards
HDFS operations.  Additionally, the token files that are created use Java serializations which
are hard/impossible to deal with in other languages. It should be replaced with a better utility
in common that can read/write protobuf-based token files, has enough flexibility to be used
with other services, and offers key functionality such as append and rename. The old version
file format should still be supported for backward compatibility, but will be effectively
> A follow-on JIRA will deprecrate fetchdt.

This message was sent by Atlassian JIRA

View raw message