hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers
Date Wed, 06 Jul 2016 22:41:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365259#comment-15365259

Inigo Goiri commented on YARN-5215:

In our internal deployment, we always reserve a buffer for the external load to spike. This
is set by tuning the available cores and memory.

[~jlowe], as you mention, we internally have preemption at both RM and NM level. We only enable
the one at NM level as it's the one with the best latency and we don't have a need for the
RM level one. As I mention in a previous comment, this patch it's just to do scheduling in
the RM, if we want to go with the full solution, we would need:
* Schedule containers considering external load in the RM
* Expose external load in the UI
* Use history to smooth external load
* Preempting containers from the RM based on external load
* Preempting containers from the NM based on external load

> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch, YARN-5215.001.patch
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is consumed by external processes and schedule based on this estimation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message