Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 31 May 2017 16:36:04 +0000 (UTC)
From: "Carlo Curino (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13071680.1494624815000.333160.1496248564213@Atlassian.JIRA>
In-Reply-To: <JIRA.13071680.1494624815000@Atlassian.JIRA>
References: <JIRA.13071680.1494624815000@Atlassian.JIRA> <JIRA.13071680.1494624815428@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6599) Support rich placement constraints
 in scheduler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 31 May 2017 16:36:08 -0000


    [ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031469#comment-16031469 ] 

Carlo Curino commented on YARN-6599:
------------------------------------

(Pardon my own delay... moving/travelling)

I hear you... 

There is a tension between an ever-growing overly-complicated scheduler code, and replication of functionalities. This, in fact, aligns with my ongoing rant about the fact that we do not have clear semantics of how all the invariants/constraints interact... if we did, it would be much easier to carve out / layer this solution. My main fear is to introduce further (complex) logic within the scheduler guts, which are already very hard to maintain. At the same time, I hear you that respecting limits/priorities/etc.. is non-trivial while sitting outside the scheduler.  

My preference would be for us to move towards a two-components solutions, where there is a planner/preprocessor that makes complex decisions using a rich language of constraints (and I mean a formal language) and a general purpose solver/compiler/optimizer type design. This would include all constraints/preferences that capture limits/priorities/etc. The second component would be a lean and mean inner-loop scheduler that only maintain states, and quickly instantiate those high level decisions into actual container placements.

I think POCs and investigations can definitely help... my only concern is to have you (or anyone else) do lots of work that we eventually decide can't be used (hence my alarmed comments). If you are ok with it, than I am all in for POCs.


> Support rich placement constraints in scheduler
> -----------------------------------------------
>
>                 Key: YARN-6599
>                 URL: https://issues.apache.org/jira/browse/YARN-6599
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org