hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
Date Mon, 09 Jun 2014 22:53:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14025879#comment-14025879
] 

Carlo Curino commented on YARN-2022:
------------------------------------

Mayank, I completely agree with you and vinod, hence my ask to Sunil to keep it simple, and
"save" AMs while respecting AM percentage (without introducing new hard-to-configure params).

This requires:
1) tagging the AM containers properly
2) sparing AMs during preemption (unless they are violating AM percentage)

On top of it, I was asking Sunil to keep an eye-out for user-limits to see whether we can
anticipate the next round of asks on preemption, which requires: 
3) during the scan of the apps within a queue to check user-limit and prioritize killing of
AMs to enforce that. 

If you prefer a smaller patch focusing on 1 and 2 but not 3, and deal with 3 in a separate
JIRA I would be ok with it. Asymptotically we would like to have preemption to enforce all
of the scheduler invariants... 

This would also allow us to play a bit more fast an loose during allocation (potentially scaling
better via batching and parallel allocations) and correct for egregious mistakes via careful
preemption (longer conversation).

> Preempting an Application Master container can be kept as least priority when multiple
applications are marked for preemption by ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-2022
>                 URL: https://issues.apache.org/jira/browse/YARN-2022
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-2022-DesignDraft.docx, Yarn-2022.1.patch
>
>
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full cluster
capacity. 
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications. In this same
scenario, map tasks from J3 and J2 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message