hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers
Date Thu, 09 Jun 2016 00:25:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321696#comment-15321696

Carlo Curino commented on YARN-5215:

[~jlowe] thanks for the comment, very useful for context, and you bring up good points on
how users "perceive" the cluster. 

[~elgoiri], correct me if I am wrong, but this feature seems ideal to "scavenge" a YARN cluster
out of otherwise utilized machines. In these settings, users should be aware that the cluster
is not constant, i.e., the effects of the fluctuations are non-trivial and expected. However,
I agree with you that surfacing them in the UI somehow is important.

All in all, I see a strong connection with over-commit, but this should be represented not
just as a heavily overcommitted cluster.  I agree with [~elgoiri] that it is useful to build
this feature in a way that more explicitly acknowledges that YARN is not the only thing running
on the cluster. 

At the same time, we should try to have a set of configurable that makes over/under-commit
appear unified and coherent to the admins, and UIs that surface them properly to users. [~elgoiri]
since you were involved in YARN-1011, can you propose a way to do that?

> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is consumed by external processes and schedule based on this estimation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message