hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7693) ContainersMonitor support configurable
Date Fri, 05 Jan 2018 17:29:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313488#comment-16313488

Miklos Szegedi commented on YARN-7693:

[~yangjiandan], than you for the detailed reply.
I understand your concern now. memory.soft_limit_in_bytes is just a good to have, AFAIK most
users disable swapping. About 6.: YARN-6677 won't hang the paused guaranteed containers but
preempt opportunistic ones first unblocking the guaranteed ones right away. This solution
was tested in a cluster by [~sandflee] in YARN-4599. The effect on guaranteed containers should
be negligible, if the logic is fast. Also the concern of promoting containers is still there
with two top level cgroups.
How do you monitor resource utilization of containers?
Do you disable OOM killer?
How would you promote?
What do you do, if the guaranteed containers grow over the opportunistic ones? They still
hang or get preempted regardless of the number and depth of top level cgroups, do not they?
Why do you reserve a gap? Does not that decrease resource utilization, which oversubscription
is supposed to fix?
bq. If the gap is less than a given value, then decrease the hard limit of Guaranteed Group.
Did you mean this? "If the gap is less than a given value, then decrease the hard limit of
Opportunistic Group." Decreasing the hard limit of the guaranteed group would mean that opportunistic
ones have an effect on guaranteed ones.

> ContainersMonitor support configurable
> --------------------------------------
>                 Key: YARN-7693
>                 URL: https://issues.apache.org/jira/browse/YARN-7693
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>            Reporter: Jiandan Yang 
>            Assignee: Jiandan Yang 
>            Priority: Minor
>         Attachments: YARN-7693.001.patch, YARN-7693.002.patch
> Currently ContainersMonitor has only one default implementation ContainersMonitorImpl,
> After introducing Opportunistic Container, ContainersMonitor needs to monitor system
metrics and even dynamically adjust Opportunistic and Guaranteed resources in the cgroup,
so another ContainersMonitor may need to be implemented. 
> The current ContainerManagerImpl ContainersMonitorImpl direct new ContainerManagerImpl,
so ContainersMonitor need to be configurable.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message