flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eron Wright (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-3670) Kerberos: Improving long-running streaming jobs
Date Tue, 29 Mar 2016 23:58:25 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217092#comment-15217092
] 

Eron Wright  edited comment on FLINK-3670 at 3/29/16 11:57 PM:
---------------------------------------------------------------

Another possibility worth considering is to leverage Hadoop's 'proxy user' functionality.
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html

In this approach, the JobManager impersonates the job submitter when accessing HDFS, HBASE,
or Hive.  Those servers would be configured to treat the JobManager principal as a superuser.

Note that the above solution isn't general, since Kafka (for example) doesn't provide proxy
user functionality.    Maybe both options could be provided.


was (Author: eronwright):
Another possibility worth considering is to leverage Hadoop's 'proxy user' functionality.
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html

In this approach, the JobManager impersonates the job submitter when accessing HDFS, HBASE,
or Hive.  Those servers would be configured to treat the JobManager principal as a proxy user.

Note that the above solution isn't general, since Kafka (for example) doesn't provide proxy
user functionality.    Maybe both options could be provided.

> Kerberos: Improving long-running streaming jobs
> -----------------------------------------------
>
>                 Key: FLINK-3670
>                 URL: https://issues.apache.org/jira/browse/FLINK-3670
>             Project: Flink
>          Issue Type: Improvement
>          Components: Command-line client, Local Runtime
>            Reporter: Maximilian Michels
>
> We have seen in the past, that Hadoop's delegation tokens are subject to a number of
subtle token renewal bugs. In addition, they have a maximum life time that can be worked around
but is very inconvenient for the user.
> As per [mailing list discussion|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Kerberos-for-Streaming-amp-Kafka-td10906.html],
a way to work around the maximum life time of DelegationTokens would be to pass the Kerberos
principal and key tab upon job submission. A daemon could then periodically renew the ticket.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message