hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7295) Support arbitrary max expiration times for delegation token
Date Wed, 29 Oct 2014 12:17:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188293#comment-14188293

Steve Loughran commented on HDFS-7295:


bq. >  pushing out new tokens from the client

bq. When the Spark Streaming app is running, it's all in the cluster. It doesn't have any
Kerberos credential at that point. I don't think it can get new tokens. Right?

the authenticated user can themselves create then push out new tokens from outside the cluster.
If an app has been deployed by management tooling, the tooling it can schedule the refesh.
The deployed app needs to export a (secured) IPC channel to received token updates in the
AM and propagate them to its containers. AFAIK, this is what Twill does. 

I've been working with my colleagues on long-lived YARN services for over a year now; token
expiry is one piece of complexity you hit. The YARN team have addressed the RM/AM token problem,
as well as the AM-launch-context token problem. For the rest, they've said "we will help you
get your keytabs over". HBase being deployed under YARN needs the same keytabs as HBase-classic;
here we actually gain from having a keytab propagation mechanism. For us we also have to add
the keytab support to the AM —but it is needed for more than just HDFS. As an example: we
also need authenticated write access to ZK when we need to update the YARN registry.  Once
you deploy applications that start needing to talk to other YARN services then the auth problem
crops up there too. Turning off token expiry for HDFS doesn't help there anyway —all it
does is set a dangerous precedent for the rest of the infrastructure. 

I guess you are disappointed by the negative feedback here: you had a simple solution to the
problem of HDFS token expiry without having to distribute keytabs. It's just that it is at
odds with the goals of kerberized-Hadoop: a secure cluster. This is precisely why those of
us who have been working on long lived YARN services for the past year haven't already actually
implemented it. Sorry.

To summarise:

h3. This is not the solution you are looking for

> Support arbitrary max expiration times for delegation token
> -----------------------------------------------------------
>                 Key: HDFS-7295
>                 URL: https://issues.apache.org/jira/browse/HDFS-7295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
> Currently the max lifetime of HDFS delegation tokens is hardcoded to 7 days. This is
a problem for different users of HDFS such as long running YARN apps. Users should be allowed
to optionally specify max lifetime for their tokens.

This message was sent by Atlassian JIRA

View raw message