hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3318) Create Initial OrderingPolicy Framework and FifoOrderingPolicy
Date Thu, 09 Apr 2015 16:37:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487631#comment-14487631

Craig Welch commented on YARN-3318:

bq. ...Do we really see non-comparator based ordering-policy. We are unnecessarily adding
two abstractions - adding policies and comparators...

In the context of the code so far, the "comparator based" approach is specific to compounding
comparators to achieve functionality (priority + fifo, fair + fifo, etc).  This was the initial
motivation for the two level api & configuration, the broader surface of the policy which
would allow for different collection types, sorting on demand, etc, (the original policy)
and the narrower one within that (comparator) for the cases where comparator logic was sufficient,
e.g. sharing a collection (for composition) and a collection type (a tree, for efficient resorting
of individual elements when required) was possible.

The two level api & configuration was not well received.  Offline Wangda has indicated
that he thinks there are policies coming up which will need the wider, initial api, with control
over the collection, sorting, etc.  Supporting policy composition for those cases would be
very awkward & is not really worth pursuing.

The various competing tradeoffs, the aversion to a multilevel api, the need for the higher
level api, and the ability to compose policies creates something of a tension, I don't think
it's realistic to try and accomplish it all together, the result will be Frankensteinian at
best.  Something has to go.  Originally, I chose the multilevel api to resolve the dilemma,
I like that choice, it seems unpopular with the crowd.  Given that, the other "optional" dynamic
is the ability to compose policies (there's no requirement for either of these as far as I
can tell, it is a "bonus feature").  While I like the composition approach, it can't be maintained
as such with the broader api and without the multilevel config/api.  As one of these has to
go and it appears it can't be the broader api or the multilevel api I suppose it will have
to be composition.  Internally there can be some composition of course, but it won't be transparent/exposed/configurable
as it was initially.

I'll put out a patch with that in a bit.

> Create Initial OrderingPolicy Framework and FifoOrderingPolicy
> --------------------------------------------------------------
>                 Key: YARN-3318
>                 URL: https://issues.apache.org/jira/browse/YARN-3318
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>         Attachments: YARN-3318.13.patch, YARN-3318.14.patch, YARN-3318.17.patch, YARN-3318.34.patch,
YARN-3318.35.patch, YARN-3318.36.patch, YARN-3318.39.patch, YARN-3318.45.patch, YARN-3318.47.patch,
> Create the initial framework required for using OrderingPolicies and an initial FifoOrderingPolicy

This message was sent by Atlassian JIRA

View raw message