hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3318) Create Initial OrderingPolicy Framework, integrate with CapacityScheduler LeafQueue supporting present behavior
Date Thu, 02 Apr 2015 06:17:53 GMT

    [ https://issues.apache.org/jira/browse/YARN-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392211#comment-14392211
] 

Craig Welch commented on YARN-3318:
-----------------------------------

[~leftnoteasy]  

SchedulerProcessEvents replaced with containerAllocated and containerReleased
Serial and SerialEpoch replaced with compareInputOrderTo(), which is the option 2 for addressing
it which we settled on offline
Added addSchedulerProcess/removeSchedulerProcess/addAllSchedulerProcesses
Changed configuration so that 
yarn.scheduler.capacity.root.default.ordering-policy=fair
will setup the fair configuration, "fifo" will setup fifo, "fair+fifo" will setup compound
fair + fifo, etc.  It is possible to setup a custom ordering policy class using a different
configuration, but the base one will handle the "friendly" setup.

[~vinodkv]
bq. It is not entirely clear how the ordering and limits work together - as one policy with
multiple facets or multiple policy types
This should be modeled as different types of policies, so that they can each focus on their
particular purpose and avoid a repetition of the intermingling which has made it difficult
to mix, match, and share capabilities.  Having multiple policy types is essential to make
it easy to combine them as needed.
bq. let's split the patch that exposes this to the client side / web UI and in the API records
into its own JIRA...premature to support this as a publicly supportable configuration...
The goal is to make this available quickly but iteratively, keeping the changes small but
making them available for use and feedback.  Clearly we can mark things "unstable", communicate
that they are not fully mature/subject to change/should be used gently, but we will need to
make it possible to activate the feature and use it in order to accomplish the use and feedback.
 We should grow it organically, gradually, iteratively, think of it is a facet of the policy
framework hooked up and available but with more to follow
bq. ...SchedulableEntity better... well, I'd actually talked [~leftnoteasy] into SchedulerProcess
:-)   So, we can chew on this a bit more & see where we go
bq. You add/remove applications to/from LeafQueue's policy but addition/removal of containers
is an event...
This has been factored differently along [~leftnoteasy]'s suggestion, it should now be consistent
bq. The notion of a comparator doesn't make sense to an admin. It is simply a policy...
Have modeled "policy configuration" differently, so "comparator" is out of sight (see above).
 
bq.  Depending on how ordering and limits come together, they may become properties of a policy
I expect them to be distinct, this is specifically an "ordering-policy", limits will be other
types of "limit-policy"(ies)

patch with these changes to follow in a few...

> Create Initial OrderingPolicy Framework, integrate with CapacityScheduler LeafQueue supporting
present behavior
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3318
>                 URL: https://issues.apache.org/jira/browse/YARN-3318
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>         Attachments: YARN-3318.13.patch, YARN-3318.14.patch, YARN-3318.17.patch, YARN-3318.34.patch,
YARN-3318.35.patch, YARN-3318.36.patch
>
>
> Create the initial framework required for using OrderingPolicies with SchedulerApplicaitonAttempts
and integrate with the CapacityScheduler.   This will include an implementation which is compatible
with current FIFO behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message