hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1015) FS should watch node resource utilization and allocate opportunistic containers if appropriate
Date Tue, 07 Nov 2017 01:14:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241281#comment-16241281

Haibo Chen commented on YARN-1015:

Thanks [~asuresh] for the review!
bq.  I don't think you should even attempt to promote the container in call. 
Right. Will remove the TODO comment to avoid confusion. 

Given our offline discussion, we could unify scheduling of reserved, guaranteed and opportunistic
That is, we can rely on the ordering of scheduling requests of reserved guaranteed, opportunistic
like we do in container increase/promotion.  Each time we allocate an opportunistic container,
we'd also add
a scheduler  request for the promotion immediately. The new request will be ordered with existing
in a FIFO fashion. At the next scheduling opportunity, after we finish allocating guaranteed
containers, we'd
check if any of the satisfied scheduling requests is for container promotion, and promote
the container explicitly.

The reason why we want to order requests in a FIFO fashion is that we want to guarantee to
users that if you
are willing to accept an OPP container for your guaranteed requests, you can start running
things early. For 
example, if you just want GUARANTEED container, you'll need to wait for  2 mins, but if you
are OK with
oversubscription, you can start right now as an OPP container and will be promoted 2 mins

> FS should watch node resource utilization and allocate opportunistic containers if appropriate
> ----------------------------------------------------------------------------------------------
>                 Key: YARN-1015
>                 URL: https://issues.apache.org/jira/browse/YARN-1015
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Haibo Chen
>         Attachments: YARN-1015-YARN-1011.00.patch, YARN-1015-YARN-1011.01.patch, YARN-1015-YARN-1011.prelim.patch
> FS should looks at resource utilization of nodes (provided by NM in heartbeat) and allocate
opportunistic containers if the resource utilization of the node is below its allocation threshold.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message