hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
Date Mon, 26 Oct 2015 06:28:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973802#comment-14973802
] 

Yi Liu edited comment on HDFS-7984 at 10/26/15 6:27 AM:
--------------------------------------------------------

{quote}
there is no way for an end user to create a token file with multiple tokens inside it, short
of building custom code to do it..
{quote}

No, I think we have.   When using existing {{Credentials#writeTokenStorageFile}}, all tokens
of the credentials will be persisted, and {{Credentials#readTokenStorageStream}} will read
all tokens too.  So what we need to do is to add different tokens to the Credentials, use
your example, there are two hdfs, we can get the delegation tokens for each of them, the {{service}}
field of the two delegation tokens should be different, we can add them to one {{Credentials}}
or through the UGI api to add them into one {{Credentials}}.
Actually even if we have multiple token files which contain only one token in each, we can
read them separately through {{Credentials#writeTokenStorageFile}}, and add them to one {{Credentials}}.

Back to the original purpose of the JIRA, I don't know why we need to specify multiple delegation
tokens in one webhdfs://,  the delegation token is used in some service to access HDFS on
behalf of user, so one hdfs only needs one delegation token for one user.  For the distcp
example you said, I think correct behavior is: user specify delegation token in each webhdfs://,
and the MR task will add the two delegation tokens to the UGI Credentials of that user.  
I think this is already supported, I have not tried the distcp on two different secured hdfs,
if there is some bug, the correct fix is as I said, it's not to support multiple delegation
tokens in one webhdfs://.
We also should not use HADOOP_TOKEN_FILE_LOCATION to solve the problem.


was (Author: hitliuyi):
{quote}
there is no way for an end user to create a token file with multiple tokens inside it, short
of building custom code to do it..
{quote}

No, I think we have.   When using existing {{Credentials#writeTokenStorageFile}}, all tokens
of the credentials will be persisted, and {{Credentials#readTokenStorageStream}} will read
all tokens too.  So what we need to do is to add different tokens to the Credentials, use
your example, there are two hdfs, we can get the delegation tokens for each of them, the {{service}}
filed of the two delegation tokens should be different, we can add them to one {{Credentials}}
or through the UGI api to add them into one {{Credentials}}.
Actually even if we have multiple token files which contain only one token in each, we can
read them separately through {{Credentials#writeTokenStorageFile}}, and add them to one {{Credentials}}.

Back to the original purpose of the JIRA, I don't know why we need to specify multiple delegation
tokens in one webhdfs://,  the delegation token is used in some service to access HDFS on
behalf of user, so one hdfs only needs one delegation token for one user.  For the distcp
example you said, I think correct behavior is: user specify delegation token in each webhdfs://,
and the MR task will add the two delegation tokens to the UGI Credentials of that user.  
I think this is already supported, I have not tried the distcp on two different secured hdfs,
if there is some bug, the correct fix is as I said, it's not to support multiple delegation
tokens in one webhdfs://.
We also should not use HADOOP_TOKEN_FILE_LOCATION to solve the problem.

> webhdfs:// needs to support provided delegation tokens
> ------------------------------------------------------
>
>                 Key: HDFS-7984
>                 URL: https://issues.apache.org/jira/browse/HDFS-7984
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: HeeSoo Kim
>            Priority: Blocker
>         Attachments: HDFS-7984.patch
>
>
> When using the webhdfs:// filesystem (especially from distcp), we need the ability to
inject a delegation token rather than webhdfs initialize its own.  This would allow for cross-authentication-zone
file system accesses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message