hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers
Date Tue, 14 Jun 2016 22:03:30 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330719#comment-15330719

Karthik Kambatla commented on YARN-5215:

We happen to have a similar low-latency framework running alongside (and occasionally on)
YARN. So, I am quite sympathetic to the problem. 

In the past, I have wondered if it makes sense to have a separate node-level agent that these
other (white-listed) services could register with to get updates on each others' usage. That
way, each framework is aware of others running on the cluster and the resources can be handed
off more gracefully. 

If we are indeed looking to steal resources from these other services, I would think those
resources should be allocated only to OPPORTUNISTIC containers and likely better handled through
YARN-1011. For instance, in your earlier example, we would actually set yarn.nodemanager.resource.memory-mb
to 14 GB which is allocated to GUARANTEED containers and YARN would also allocate OPPORTUNISTIC
containers upto 2 GB based on how much of it is used by other frameworks. 

And, as Jason was mentioning earlier (IIUC), YARN-5202 provides this without the support for
special OPPORTUNISTIC containers. Am I missing something? 

> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch, YARN-5215.001.patch
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is consumed by external processes and schedule based on this estimation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message