hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
Date Wed, 31 May 2017 16:36:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031469#comment-16031469

Carlo Curino commented on YARN-6599:

(Pardon my own delay... moving/travelling)

I hear you... 

There is a tension between an ever-growing overly-complicated scheduler code, and replication
of functionalities. This, in fact, aligns with my ongoing rant about the fact that we do not
have clear semantics of how all the invariants/constraints interact... if we did, it would
be much easier to carve out / layer this solution. My main fear is to introduce further (complex)
logic within the scheduler guts, which are already very hard to maintain. At the same time,
I hear you that respecting limits/priorities/etc.. is non-trivial while sitting outside the

My preference would be for us to move towards a two-components solutions, where there is a
planner/preprocessor that makes complex decisions using a rich language of constraints (and
I mean a formal language) and a general purpose solver/compiler/optimizer type design. This
would include all constraints/preferences that capture limits/priorities/etc. The second component
would be a lean and mean inner-loop scheduler that only maintain states, and quickly instantiate
those high level decisions into actual container placements.

I think POCs and investigations can definitely help... my only concern is to have you (or
anyone else) do lots of work that we eventually decide can't be used (hence my alarmed comments).
If you are ok with it, than I am all in for POCs.

> Support rich placement constraints in scheduler
> -----------------------------------------------
>                 Key: YARN-6599
>                 URL: https://issues.apache.org/jira/browse/YARN-6599
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message