hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers
Date Tue, 14 Jun 2016 21:26:30 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330649#comment-15330649

Inigo Goiri commented on YARN-5215:

[~kasha], in our use case we are targeting co-locating with latency sensitive workloads and
they have diurnal patterns. For this type of workload, we need to be fairly reactive. Actually,
preempting containers at the NM following the {{ContainersMonitor}} loop would be ideal.

The improvements in utilization are significant as right now we are just reserving for the
peak of the latency sensitive workloads (around ~50%) of the machine. We tried at some point
to have a separate service to periodically change the resources of the NMs but it's harder
to operate.

In any case, in this first patch, we are just preventing scheduling containers and not adding
preemption. I can add the following changes to the current patch:
# UI improvements
# History in the utilization to take decisions
# Preempting containers from the RM
# Preempting containers from the NM

The problem with preemption is that we would go into what to preempt and that might have some
dependencies in the opportunistic stuff in YARN-1011.

> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch, YARN-5215.001.patch
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is consumed by external processes and schedule based on this estimation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message