hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
Date Wed, 08 Jun 2016 18:04:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321091#comment-15321091
] 

Carlo Curino commented on YARN-5215:
------------------------------------

This sounds a generally good idea (patch obviously need work). It would be able for example
to take into account of HDFS resource consumption, 
or more generally if other services are run on the same box and have non-constant resource
utilization we would not need to pessimistically bound the 
resources given to YARN. 

Questions:
 # How do we ensure that there are no weird feedback loops, e.g., a task is schedule and consume
lots of resources, and as a consequence the scheduler
lower the load on the node, and this task graphs even more resources? For CPU/Mem we might
rely on enforcement, but what about adding non-enforced resources?
 # Would you also trigger preemption based on this? Or only avoid scheduling more load if
the node is busy? 
 # What is the interplay between this and the work on Overcommit? 
 # Patch looks very simple/small for this feature, is that all there is needed here? More
dependencies?
 # How do we test this till we are convinced works? (are you using it anywhere?)

[~kasha], [~kkaranasos], [~asuresh] can you guys comment on this?

> Scheduling containers based on load in the servers
> --------------------------------------------------
>
>                 Key: YARN-5215
>                 URL: https://issues.apache.org/jira/browse/YARN-5215
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Inigo Goiri
>         Attachments: YARN-5215.000.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the resources.
The proposal is to use the utilization information in the node and the containers to estimate
how much is actually available in the NMs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message