hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2889) Limit in the number of opportunistic container requests per AM
Date Wed, 29 Nov 2017 19:56:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271444#comment-16271444

Arun Suresh commented on YARN-2889:

bq. I don't think you can expect user to respect this limit, at least not for the first time.
On the other hand, number of container requests is vary against the size of the job, how you
gonna define this limit? Therefore, it doesn't seem to be practical to me.
Apologize - but am not sure I follow the argument.
So the limit currently is something we enforce at a per-AM-per-allocate call level. If an
AM asks for more O containers, the requests are queued in the RM (in the application's {{OpportunisticContainerContext}})
and will be satisfied in subsequent allocate calls.

bq. If NM queue is full, can we avoid assigning any O containers to that node? That means
when preparing top K least loaded nodes, we need to exclude nodes whose queue is already full.
Agreed - that is a good idea. Something we have been planning on doing actually. Feel free
to raise a JIRA - will help review. To be honest, instead of a total sort, a partial sorting
of nodes which includes only nodes whose queue length < x, where x is a small value to
give a higher probably for the O container to run - might be useful.

bq. Each queue size is limited, so I don't see why lots of O containers would flood the system.
Hmmm.. so it is still possible for queues to be filled with O containers from a single AM
- thereby denying other AMs for getting O containers. YARN-7258 handles this somewhat, by
spreading the O containers requested by an AM across multiple allocate calls - so that other
AMs get a chance.

bq. You can't say an AM is malicious if it requests only opportunistic containers (too many).
Unless this was the design, then you need to setup correct user expectation with some document,
and explain what is the correct user case.
Understand - which is why we are interested in auto-limiting the number of O containers allocated
to an AM. Another option is to say - the AM does not explicitly ask for O containers and the
RM will allocate an O container based on queue/cluster capacity or number of nodes with 0
queue length etc.

> Limit in the number of opportunistic container requests per AM
> --------------------------------------------------------------
>                 Key: YARN-2889
>                 URL: https://issues.apache.org/jira/browse/YARN-2889
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Konstantinos Karanasos
>            Assignee: Arun Suresh
> We introduce a way to limit the number of queueable requests that each AM can submit
to the LocalRM.
> This way we can restrict the number of queueable containers handed out by the system,
as well as throttle down misbehaving AMs (asking for too many queueable containers).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message