hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2889) Limit in the number of opportunistic container requests per AM
Date Wed, 29 Nov 2017 05:35:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270181#comment-16270181

Arun Suresh commented on YARN-2889:

Thanks for your interest in this JIRA [~cheersyang].
We have actually implemented a bit of this as part of YARN-7258. The basic idea is that -
originally, if an AM asks for x containers in an allocate call, the {{OpportunisticContainerAllocator}}
would respond with x containers. The containers would be distributed (in a simple round-robin
manner) across the top K least loaded nodes. The problem with this is that if there are a
lot of applications running and all of them asking for O containers - or if AMs indiscriminately
ask for too many O containers in a single call, There would be too many O containers allocated
and most of them will get pre-emempted - leading to a lot of wasted work.
In YARN-7258, we inserted a restriction that in a single allocate call, the AM will get a
max of K containers (where K is the top-K least loaded nodes - In addition we also try to
allocate a container on the same rack, if a request node is not present in the top K - or
on the least loaded node if no node from the same rack is found in the top K). The remaining
requests are queued and retried in the next allocate call (with possibly a new top-K list
- if enough time has passed).

> Limit in the number of opportunistic container requests per AM
> --------------------------------------------------------------
>                 Key: YARN-2889
>                 URL: https://issues.apache.org/jira/browse/YARN-2889
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Konstantinos Karanasos
>            Assignee: Arun Suresh
> We introduce a way to limit the number of queueable requests that each AM can submit
to the LocalRM.
> This way we can restrict the number of queueable containers handed out by the system,
as well as throttle down misbehaving AMs (asking for too many queueable containers).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message