hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-3053) [Security] Review and implement authentication in ATS v.2
Date Thu, 26 Jan 2017 18:34:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840182#comment-15840182
] 

Varun Saxena edited comment on YARN-3053 at 1/26/17 6:34 PM:
-------------------------------------------------------------

[~jianhe], first of all whatever I mentioned above is based on how I think off-app clients
would work. We haven't yet finalized and brainstormed design of off-app clients because its
not in immediate scope. For app collectors I agree with you that we do not need to store tokens
in state store. 
By off-app clients I mean frameworks like Oozie or Hive publishing their entities i.e. some
info before and after executing a workflow or a set of queries which does not necessarily
fit within the scope of individiual AMs'. In other words, information which spans multiple
YARN applications and hence cannot be published by individual AMs'.

For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual
AMs'. So most probably clients will have to get this info from YARN (first when client comes
up and later on connection loss if collector goes down). This query will most probably go
to RM.  And this communication needs to be kerberos authenticanted. We can probably open up
a channel to pull a token as well from RM (after off-app collector reports it to RM via NM).
But an easier way would be to get it from collector via an explicit get delegation token API.
Because such clients would anyways need to do kerberos authentication.
If we do so, we may need to store and recover tokens.
Anyways off-app clients require a little more thought. I can think of ways of avoiding storing
token even in this case but that may make other parts more complex. Let us discuss and handle
use-cases of off-app clients once we start working on them.

bq. I'm not sure the original collector design had accounted for unmanaged AM in general case
Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container
is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121
for that.


was (Author: varun_saxena):
[~jianhe], first of all whatever I mentioned above is based on how I think off-app clients
would work. For app collectors I agree with you that we do not need to store tokens in state
store. We haven't yet finalized and brainstormed design of off-app clients because its not
in immediate scope. By off-app clients I mean frameworks like Oozie or Hive publishing their
entities i.e. some info before and after executing a workflow or a set of queries which does
not necessarily fit within the scope of individiual AMs'. In other words, information which
spans multiple YARN applications and hence cannot be published by individual AMs'.

For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual
AMs'. So most probably clients will have to get this info from YARN (first when client comes
up and later on connection loss if collector goes down). This query will most probably go
to RM.  And this communication needs to be kerberos authenticanted. We can probably open up
a channel to pull a token as well from RM (after off-app collector reports it to RM via NM).
But an easier way would be to get it from collector via an explicit get delegation token API.
Because such clients would anyways need to do kerberos authentication.
If we do so, we may need to store and recover tokens.
Anyways off-app clients require a little more thought. I can think of ways of avoiding storing
token even in this case but that may make other parts more complex. Let us discuss and handle
use-cases of off-app clients once we start working on them.

bq. I'm not sure the original collector design had accounted for unmanaged AM in general case
Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container
is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121
for that.

> [Security] Review and implement authentication in ATS v.2
> ---------------------------------------------------------
>
>                 Key: YARN-3053
>                 URL: https://issues.apache.org/jira/browse/YARN-3053
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Varun Saxena
>              Labels: YARN-5355, yarn-5355-merge-blocker
>         Attachments: ATSv2Authentication(draft).pdf
>
>
> Per design in YARN-2928, we want to evaluate and review the system for security, and
ensure proper security in the system.
> This includes proper authentication, token management, access control, and any other
relevant security aspects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message