hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers
Date Wed, 15 Jun 2016 04:02:09 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331102#comment-15331102

Carlo Curino commented on YARN-5215:

Regarding preemptable vs OPPORTUNISTIC, we had this conversation with [~kkaranasos] and [~asuresh],
where containers types could be {{NON-PREEMPTABLE}}, {{PREEMPTABLE}}, {{OPPORTUNISTIC}}, describing
basically increasing level of how likely is a task to be interrupted. 
For {{NON-PREEMPTABLE}} the system does everything it can not to interrupt (bar physical machine
failures), {{PREEMPTABLE}} are containers with a low, but non-null chance of being interrupted
by the scheduler (e.g., allocations above a queue capacity, but on dedicated resources), and
{{OPPORTUNISTIC}} are tasks that have a high(er) chance of being interrupted as they run on
the left-over capacity from other containers, or on overcommitted resources or, or other risky
forms of resource harvesting (as in this JIRA).   

BTW [~elgoiri] I would suggest to turn this in an umbrella JIRA, and separate the sub-step
you list [in the comment above | https://issues.apache.org/jira/browse/YARN-5215?focusedCommentId=15330649&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15330649].
Some are likely less controversial and can be settled/committed earlier, while other parts
are too "interesting" to go in easily :-)

> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch, YARN-5215.001.patch
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is consumed by external processes and schedule based on this estimation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message