hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers
Date Wed, 08 Jun 2016 23:55:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321672#comment-15321672

Inigo Goiri commented on YARN-5215:

Yes, I realized that the original title didn't mention external load. Fixed now, sorry about
that; I think it's more clear. Feel free to tweak the description more.

As you mention, we could achieve this by tweaking the "guaranteed" size. However, I think
that having the explicit concept regarding external utilization makes it simpler and it's
compatible with the overcommit approach (both can be enabled/disabled independently). In addition,
the concept of node utilization is not planned to be used in YARN-1011 for now.

I'm going to post during the next hour a patch with:
* Unit tests
* Conf switches
* Boundary checks

Then, I agree that we need to report this properly to the user. I was thinking on exposing
the {{getExternalUtilization()}} or the updated {{getUnallocated()}} through the Web UI, etc.
If we decide this feature should go ahead, I would add here or in a new JIRA.

To summarize the issues to discuss/finalize are:
* Decide if this should be a separate feature or within overcommit
* Add unit tests
* Add conf switches
* Add boundary checks
* Interface to expose this information

Regarding YARN-5202 vs YARN-1011, it looks to me like there's a lot of overlap between them.
I think it'd be better to port most of YARN-5202 into YARN-1011. We probably should move this
discussion into one of them.

> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is consumed by external processes and schedule based on this estimation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message