hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Yang (Jira)" <j...@apache.org>
Subject [jira] [Updated] (YARN-10854) Support marking inactive node as untracked without configured include path
Date Thu, 15 Jul 2021 10:33:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tao Yang updated YARN-10854:
----------------------------
    Description: 
Currently inactive nodes which have been decommissioned/shutdown/lost for a while(specified
expiration time defined via {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60
seconds by default) and not exist in both include and exclude files can be marked as untracked
nodes and can be removed from RM state (YARN-4311). It's very useful when auto-scaling is
enabled in elastic cloud environment, which can avoid unlimited increase of inactive nodes
(mostly are decommissioned nodes).

But this only works when the include path is configured, mismatched for most of our cloud
environments without configured white list of nodes, which can lead to easily control for
the auto-scaling of nodes without further security requirements.

So I propose to support marking inactive node as untracked without configured include path,
to be compatible with the former versions, we can add a switch config for this.

Any thoughts/suggestions/feedbacks are welcome!

  was:
Currently inactive nodes which have been decommissioned/shutdown/lost for a while(specified
expiration time defined via {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60
seconds by default) and not exist in both include and exclude files can be marked as untracked
nodes and can be removed from RM state. It's very useful when auto-scaling is enabled in elastic
cloud environment, which can avoid unlimited increase of inactive nodes (mostly are decommissioned
nodes).

But this only works when the include path is configured, mismatched for most of our cloud
environments without configured white list of nodes, which can lead to easily control for
the auto-scaling of nodes without further security requirements.

So I propose to support marking inactive node as untracked without configured include path,
to be compatible with the former versions, we can add a switch config for this.

Any thoughts/suggestions/feedbacks are welcome!


> Support marking inactive node as untracked without configured include path
> --------------------------------------------------------------------------
>
>                 Key: YARN-10854
>                 URL: https://issues.apache.org/jira/browse/YARN-10854
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>            Priority: Major
>         Attachments: YARN-10854.001.patch
>
>
> Currently inactive nodes which have been decommissioned/shutdown/lost for a while(specified
expiration time defined via {{yarn.resourcemanager.node-removal-untracked.timeout-ms}}, 60
seconds by default) and not exist in both include and exclude files can be marked as untracked
nodes and can be removed from RM state (YARN-4311). It's very useful when auto-scaling is
enabled in elastic cloud environment, which can avoid unlimited increase of inactive nodes
(mostly are decommissioned nodes).
> But this only works when the include path is configured, mismatched for most of our cloud
environments without configured white list of nodes, which can lead to easily control for
the auto-scaling of nodes without further security requirements.
> So I propose to support marking inactive node as untracked without configured include
path, to be compatible with the former versions, we can add a switch config for this.
> Any thoughts/suggestions/feedbacks are welcome!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message