Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 4 Aug 2017 15:24:00 +0000 (UTC)
From: "Varun Saxena (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13038882.1485779131000.94710.1501860240411@Atlassian.JIRA>
In-Reply-To: <JIRA.13038882.1485779131000@Atlassian.JIRA>
References: <JIRA.13038882.1485779131000@Atlassian.JIRA> <JIRA.13038882.1485779131786@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (YARN-6133) [ATSv2 Security] Renew
 delegation token for app automatically if an app collector is active
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 04 Aug 2017 15:24:05 -0000


    [ https://issues.apache.org/jira/browse/YARN-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114248#comment-16114248 ] 

Varun Saxena edited comment on YARN-6133 at 8/4/17 3:23 PM:
------------------------------------------------------------

Thanks [~rohithsharma] for the review.
bq. Token is renewed just before 10 seconds. Should it be increased?
What do you suggest? 10 seconds should be enough as we renew only in DT manager i.e. internally in NM. Token doesn't need to go to AM. So 10 seconds is more than enough. Right?

bq. TimelineCollectorManager has introduced synchronized block. This is not necessary right.?
This is to avoid race between Collector stopping and renewal timer expiring. So that additional renewal timer is not set unnecessarily.  Has no functional impact though even if we set because it just wont find collector on expiry. But I thought better to avoid it altogether. Thoughts?

bq. Renewer threads count is 1. Given load on NM not much, one thread can renew it. But I would suggest to keep it to 50?
How many active collectors do we expect in one NM? Token renewal and token generation is not a very heavy task as well. Assuming we have 1000 active apps in say a 5000 node large cluster, we will have AMs' distributed across multiple nodes. So It is unlikely you will have more than 4-5 app collectors running in any NM at a particular moment. And even there it is unlikely that all collectors will have their token renewal expiry at same moment.
There are no guarantees though. But it is unlikely. We may have a situation wherein we launch AMs' on a particular node partition though. In this case there might be some hotspotting, as in multiple app collectors on one node.
But even there, 50 might be too many I think. We can keep a value higher than 1 though if you have concerns with only 1 thread, maybe 3-5. Keep it configurable with default 3 or 5?


was (Author: varun_saxena):
Thanks [~rohithsharma] for the review.
bq. Token is renewed just before 10 seconds. Should it be increased?
What do you suggest? 10 seconds should be enough as we renew only in DT manager i.e. internally in NM. Token doesn't need to go to AM. Right?

bq. TimelineCollectorManager has introduced synchronized block. This is not necessary right.?
This is to avoid race between Collector stopping and renewal timer expiring. So that additional renewal timer is not set unnecessarily.  Has no functional impact though even if we set because it just wont find collector on expiry. But I thought better to avoid it altogether. Thoughts?

bq. Renewer threads count is 1. Given load on NM not much, one thread can renew it. But I would suggest to keep it to 50?
How many active collectors do we expect in one NM? Token renewal and token generation is not a very heavy task as well. Assuming we have 1000 active apps in say a 5000 node large cluster, we will have AMs' distributed across multiple nodes. So It is unlikely you will have more than 4-5 app collectors running in any NM at a particular moment. And even there it is unlikely that all collectors will have their token renewal expiry at same moment.
There are no guarantees though. But it is unlikely. We may have a situation wherein we launch AMs' on a particular node partition though. In this case there might be some hotspotting, as in multiple app collectors on one node.
But even there, 50 might be too many I think. We can keep a value higher than 1 though if you have concerns with only 1 thread, maybe 3-5. Keep it configurable with default 3 or 5?

> [ATSv2 Security] Renew delegation token for app automatically if an app collector is active
> -------------------------------------------------------------------------------------------
>
>                 Key: YARN-6133
>                 URL: https://issues.apache.org/jira/browse/YARN-6133
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-5355-merge-blocker
>         Attachments: YARN-6133-YARN-5355.01.patch, YARN-6133-YARN-5355.02.patch, YARN-6133-YARN-5355.03.patch
>
>


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org