hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sriram Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2877) Extend YARN to support distributed scheduling
Date Sat, 22 Nov 2014 15:21:36 GMT

    [ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221991#comment-14221991

Sriram Rao commented on YARN-2877:

[~leftnoteasy] By definition, the allocation decisions made by the central RM win out.  That
is, whenever there is a conflict, *guaranteed-start* (or CONSERVATIVE) containers will be
executed prior to *queueable* (or OPTIMISTIC) containers.  This could also means that the
NM may be forced to preempt running *queueable* containers to make room.   Lastly, to allow
some level of predictability in terms of execution time for *queueable* containers, we could
use leases---a *queueable* container is allowed to execute for at most N secs even when there
is conflict and if the container hasn't exited, the NM will preempt them after that time interval
elapses (i.e., lease expires).  This mechanism can allow minimizing preemption for *queueable*

Re: your other questions:
# Capacity is enforced for *guaranteed-start* containers.  For *queueable* containers, policies
could be pushed down from central-RM ([YARN-2885|https://issues.apache.jira/browse/YARN-2885])
# It is not necessary that the *queueable* containers factor into central RM's allocation
choices.  That said, having that information at the central-RM can help minimize preemption.
# For enabling load balancing of queues at the NM's (([YARN-2888|https://issues.apache.jira/browse/YARN-2888]),
allow AM's to make choices of where to submit *queueable* containers ([YARN-2887|https://issues.apache.jira/browse/YARN-2887]),
exposing queue information to local-RM's is desirable.

> Extend YARN to support distributed scheduling
> ---------------------------------------------
>                 Key: YARN-2877
>                 URL: https://issues.apache.org/jira/browse/YARN-2877
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
> This is an umbrella JIRA that proposes to extend YARN to support distributed scheduling.
 Briefly, some of the motivations for distributed scheduling are the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise idle resources
on individual machines.
> 2. Reduce allocation latency.  Tasks where the scheduling time dominates (i.e., task
execution time is much less compared to the time required for obtaining a container from the

This message was sent by Atlassian JIRA

View raw message