hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6133) [ATSv2 Security] Renew delegation token for app automatically if an app collector is active
Date Fri, 04 Aug 2017 15:24:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114248#comment-16114248
] 

Varun Saxena edited comment on YARN-6133 at 8/4/17 3:23 PM:
------------------------------------------------------------

Thanks [~rohithsharma] for the review.
bq. Token is renewed just before 10 seconds. Should it be increased?
What do you suggest? 10 seconds should be enough as we renew only in DT manager i.e. internally
in NM. Token doesn't need to go to AM. So 10 seconds is more than enough. Right?

bq. TimelineCollectorManager has introduced synchronized block. This is not necessary right.?
This is to avoid race between Collector stopping and renewal timer expiring. So that additional
renewal timer is not set unnecessarily.  Has no functional impact though even if we set because
it just wont find collector on expiry. But I thought better to avoid it altogether. Thoughts?

bq. Renewer threads count is 1. Given load on NM not much, one thread can renew it. But I
would suggest to keep it to 50?
How many active collectors do we expect in one NM? Token renewal and token generation is not
a very heavy task as well. Assuming we have 1000 active apps in say a 5000 node large cluster,
we will have AMs' distributed across multiple nodes. So It is unlikely you will have more
than 4-5 app collectors running in any NM at a particular moment. And even there it is unlikely
that all collectors will have their token renewal expiry at same moment.
There are no guarantees though. But it is unlikely. We may have a situation wherein we launch
AMs' on a particular node partition though. In this case there might be some hotspotting,
as in multiple app collectors on one node.
But even there, 50 might be too many I think. We can keep a value higher than 1 though if
you have concerns with only 1 thread, maybe 3-5. Keep it configurable with default 3 or 5?


was (Author: varun_saxena):
Thanks [~rohithsharma] for the review.
bq. Token is renewed just before 10 seconds. Should it be increased?
What do you suggest? 10 seconds should be enough as we renew only in DT manager i.e. internally
in NM. Token doesn't need to go to AM. Right?

bq. TimelineCollectorManager has introduced synchronized block. This is not necessary right.?
This is to avoid race between Collector stopping and renewal timer expiring. So that additional
renewal timer is not set unnecessarily.  Has no functional impact though even if we set because
it just wont find collector on expiry. But I thought better to avoid it altogether. Thoughts?

bq. Renewer threads count is 1. Given load on NM not much, one thread can renew it. But I
would suggest to keep it to 50?
How many active collectors do we expect in one NM? Token renewal and token generation is not
a very heavy task as well. Assuming we have 1000 active apps in say a 5000 node large cluster,
we will have AMs' distributed across multiple nodes. So It is unlikely you will have more
than 4-5 app collectors running in any NM at a particular moment. And even there it is unlikely
that all collectors will have their token renewal expiry at same moment.
There are no guarantees though. But it is unlikely. We may have a situation wherein we launch
AMs' on a particular node partition though. In this case there might be some hotspotting,
as in multiple app collectors on one node.
But even there, 50 might be too many I think. We can keep a value higher than 1 though if
you have concerns with only 1 thread, maybe 3-5. Keep it configurable with default 3 or 5?

> [ATSv2 Security] Renew delegation token for app automatically if an app collector is
active
> -------------------------------------------------------------------------------------------
>
>                 Key: YARN-6133
>                 URL: https://issues.apache.org/jira/browse/YARN-6133
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-5355-merge-blocker
>         Attachments: YARN-6133-YARN-5355.01.patch, YARN-6133-YARN-5355.02.patch, YARN-6133-YARN-5355.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message