hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2889) Limit in the number of opportunistic container requests per AM
Date Wed, 29 Nov 2017 10:58:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270591#comment-16270591

Weiwei Yang commented on YARN-2889:

Thanks [~asuresh], [~kkaranasos] for the quick response.
I don't think you can expect user to respect this limit, at least not for the first time.
On the other hand, number of container requests is vary against the size of the job, how you
gonna define this limit? Therefore, it doesn't seem to be practical to me. 

Some other thoughts might be related,
# If NM queue is full, can we avoid assigning any O containers to that node? That means when
preparing top K least loaded nodes, we need to exclude nodes whose queue is already full.
# Each queue size is limited, so I don't see why lots of O containers would flood the system.
# You can't say an AM is malicious if it requests only opportunistic containers (too many).
Unless this was the design, then you need to setup correct user expectation with some document,
and explain what is the correct user case.

Well these comments are based on my current understanding by reading JIRA comments and design
doc, please correct me if anything goes wrong. Thanks!

> Limit in the number of opportunistic container requests per AM
> --------------------------------------------------------------
>                 Key: YARN-2889
>                 URL: https://issues.apache.org/jira/browse/YARN-2889
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Konstantinos Karanasos
>            Assignee: Arun Suresh
> We introduce a way to limit the number of queueable requests that each AM can submit
to the LocalRM.
> This way we can restrict the number of queueable containers handed out by the system,
as well as throttle down misbehaving AMs (asking for too many queueable containers).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message