hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Wed, 27 Jan 2016 17:20:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119825#comment-15119825
] 

Karthik Kambatla commented on YARN-1011:
----------------------------------------

[~nroberts] - interesting ideas. The notion of two schedulers definitely frees us from the
confines we have today. 

How about doing the following on node update (along the lines of two scheduler suggestion):
{code}
process_newly_launched_containers();
process_completed_containers();
while (guaranteed_resources_on_node_are_still_available) {
  app = pickApp();
  if (app.hasOpportunisticContainersRunningOnTheNode()) {
    promote_container_to_guaranteed();
  } else {
    allocate_guaranteed_container_as_we_do_today(); // includes reservation
  }
}
while (opportunistic_resources_on_node_are_still_available) {
  app = pickAppOkayWithOpportunisticContainers();
  allocate_opportunistic_container(); // reservations are not allowed
}
{code}

This way:
# Apps that can't tolerate using opportunistic containers can state their preference to have
only guaranteed containers.
# For apps that are okay with opportunistic containers, a container is promoted only when
the app deserves guaranteed containers. 

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message