hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7403) Compute global and local preemption
Date Tue, 31 Oct 2017 20:43:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227507#comment-16227507
] 

Carlo Curino commented on YARN-7403:
------------------------------------

The attached screenshot shows an example of "globally" calculated local preemption. In particular,
it tries to highlight the problem of locality of the demand vs availability of preemptable
containers. The code in draf3 patch computes the total preemption to be 100 containers, it
splits it among SC1 and SC2 based on B demand (so 66/33) and cap the preemption by the number
of preemptable containers in A1 which is 20 in SC2.   

Other "splitting" decisions can be made, enforcing different invariants, e.g., that all 100
containers are preempted etc... I think the current  policy is reasonable, when combined with
a stateful AMRMPRoxy policy that "relax" locality demand, as the requests from B will eventually
be migrated towards the sub-cluster where demand is being fulfilled, i.e., in a later time
B's demand should be in SC1 and more preemptiong of A1 containers in SC1 should kick in. 

Thoughts?



> Compute global and local preemption
> -----------------------------------
>
>                 Key: YARN-7403
>                 URL: https://issues.apache.org/jira/browse/YARN-7403
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: federation
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-7403.draft.patch, YARN-7403.draft2.patch, YARN-7403.draft3.patch,
global-queues-preemption.PNG
>
>
> This JIRA tracks algorithmic effort to combine the local queue views of capacity guarantee/use/demand
and compute the global amount of preemption, and based on that, "where" (in which sub-cluster)
preemption will be enacted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message