hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4009) WebHdfsFileSystem and HftpFileSystem don't need delegation tokens
Date Wed, 10 Oct 2012 18:57:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473472#comment-13473472

Karthik Kambatla commented on HDFS-4009:

Thanks Owen.

If I understand you correctly, in addition to letting user run commands/jobs as another user,
delegation tokens also act as a backup authentication mechanism, should the kerberos ticket
time out.

If that is the case, I see why we need to renew delegation tokens. In that case, we need to
decide if each FileSystem should have its own (instance of) {{DelegationTokenRenewer}} or
if we should have a single {{DelegationTokenRenewerService}} that FileSystems can register/de-register.
To me, it seems reasonable to have a single service for different clients, by augmenting the
current {{DelegationTokenRenwer}}.

In my previous comment in the JIRA, I propose the following to that effect:
# Create a new abstract class TokenManager that holds a list (queue) of tokens, their kind,
renewal period, a TokenRenewer implementation, and an abstract method #manage() to manage
(renew/re-fetch/cancel) the tokens as appropriate.
# Each FileSystem (WebHdfs and Hftp) has its own implemenation (local extended class) of the
# DelegationTokenRenewer should be a singleton.
# DelegationTokenRenewer allows registering and de-registering TokenManagers. FileSystems
call register() at init(), and de-register() at close().
# DelegationTokenRenewer stores the registered TokenManagers in its DelayQueue. On a TokenManager's
turn, the manage() method is invoked. If the manage fails, the TokenManager is automatically

Can you please weigh-in on this approach?

> WebHdfsFileSystem and HftpFileSystem don't need delegation tokens
> -----------------------------------------------------------------
>                 Key: HDFS-4009
>                 URL: https://issues.apache.org/jira/browse/HDFS-4009
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.0.0-alpha
>            Reporter: Tom White
>            Assignee: Karthik Kambatla
>         Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch
> Parent JIRA to track the work of removing delegation tokens from these filesystems. 
> This JIRA has evolved from the initial issue of these filesystems not stopping the DelegationTokenRenewer
thread they were creating.
> After further investigation, Daryn pointed out - "If you can get a token, you don't need
a token"! Hence, these filesystems shouldn't use delegation tokens.
> Evolution of the JIRA is listed below:
> Update 2:
> DelegationTokenRenewer is not required. The filesystems that are using it already have
Krb tickets and do not need tokens. Remove DelegationTokenRenewer and all the related logic
from WebHdfs and Hftp filesystems.
> Update1:
> DelegationTokenRenewer should be Singleton - the instance and renewer threads should
be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop
the renewer, and only register/de-register for token renewal.
> Initial issue:
> HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when
they are closed. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message