hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
Date Thu, 26 Jun 2014 10:05:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044535#comment-14044535
] 

Sunil G commented on YARN-2022:
-------------------------------

Hi [~mayank_bansal] and [~leftnoteasy]

I was checking few scenarios with these two options.

As Wangda mentioned, getAbsoluteMaximumCapacity is used for calculating total #application
within a queue. With current preemption mechanism, each application will be completely preempted
and hence there arises no scenario that AMs will be occupy a big part of cluster. But with
saving AM feature w.r.t maxAMCapacity (and *getAbsoluteMaximumCapacity()*), I feel there can
be a scenario where major portion of cluster going for AMs. 

A scenario with QueueA (with 20% capacity and 100% maximum), QueueB (with 80% capacity and
100% maximum) and 0.5 as AM percentage. Assume 100% cluster is taken over by QueueA jobs with
50% of AMs (different users). When demand is there from QueueB, QueueA will end up with 50%
of AMs (saving AMs w.r.t maxAMCapacity). Now QueueB will start running jobs and it can also
take 25% of AMs. So there are now more chances of having more AMs in cluster compared to earlier
design.

May be I feel we can use *getAbsoluteCapacity* for now (seems more safer) and as Wangda mentioned
we can discuss in another JIRA about the other corner cases w.r.t preemption and CapacityScheduler
configurations. Please suggest your thoughts w.r.t this scenario.


> Preempting an Application Master container can be kept as least priority when multiple
applications are marked for preemption by ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-2022
>                 URL: https://issues.apache.org/jira/browse/YARN-2022
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, YARN-2022.3.patch,
YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, YARN-2022.7.patch, Yarn-2022.1.patch
>
>
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full cluster
capacity. 
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications. In this same
scenario, map tasks from J3 and J2 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message