hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4902) [Umbrella] Generalized and unified scheduling-strategies in YARN
Date Fri, 05 Aug 2016 23:56:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410294#comment-15410294
] 

Wangda Tan commented on YARN-4902:
----------------------------------

[~kkaranasos],

bq. LRA planning is much more than an implementation. Think of it as planning multiple applications
at once. This is something that the scheduler cannot do, no matter what its implementation
is...
I know the implementation of YARN-1051, I participated reviewing JIRAs of YARN-1051 from the
beginning.
I would say we should have a unified API for client to use instead of continue adding new
APIs. Breaking down APIs to different sets is especially bad for new feature incubation, this
is one of the major reason we created YARN-4902. Planning more than one applications at once
should be an enhancement inside scheduler instead of something visible by end user.
And I'm not sure if we should create a new LRA planner for that, if you're thinking to implement
it using approach of YARN-1051, you may need to reserve a chunk of resources for "LRA scheduling".
This approach may add additional latency to scheduling (since heavier computation) and reduce
throughput (since additional resources need to be reserved). But this approach looks more
reasonable while doing YARN-1051 -- it is straightforward reserve some resources for reservation
system.
I also have mentioned partial-global-scheduling vs. global-optimal-scheduling in design doc
of YARN-5139: https://issues.apache.org/jira/secure/attachment/12822180/YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf.
You can take a look at it if you're interested.
I can understand your proposal may look different from my guess above, we can discuss more
once you have a more concrete design for that.
 
bq. I would say that it is the other way around..
I'm not care too much about if we should support cardinality via GUTS API or support anti-affinity
via cardinality syntaxes. We should choose a more generic/extensible API which can support
both.

And I just created a branch YARN-4902 and created sub task YARN-5478, we can discuss more
about Java API definition in that JIRA.

> [Umbrella] Generalized and unified scheduling-strategies in YARN
> ----------------------------------------------------------------
>
>                 Key: YARN-4902
>                 URL: https://issues.apache.org/jira/browse/YARN-4902
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Wangda Tan
>         Attachments: Generalized and unified scheduling-strategies in YARN -v0.pdf, LRA-scheduling-design.v0.pdf,
YARN-5468.prototype.patch
>
>
> Apache Hadoop YARN's ResourceRequest mechanism is the core part of the YARN's scheduling
API for applications to use. The ResourceRequest mechanism is a powerful API for applications
(specifically ApplicationMasters) to indicate to YARN what size of containers are needed,
and where in the cluster etc.
> However a host of new feature requirements are making the API increasingly more and more
complex and difficult to understand by users and making it very complicated to implement within
the code-base.
> This JIRA aims to generalize and unify all such scheduling-strategies in YARN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message