hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
Date Fri, 14 Oct 2016 19:19:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576235#comment-15576235
] 

Eric Payne commented on YARN-2009:
----------------------------------

[~sunilg], I have one concern about how {{TempAppPerPartition#toBePreempted}} is calculated.
Please consider the use case of where one low priority app is taking up all resources and
a second high priority app is trying to get started:

_NOTE; I am ignoring {{userLimitResource}} and {{idealAssignedForUser}} because in this use
case there is only 1 user. Ignoring {{app.selected}} as well._
{code:title=FifoIntraQueuePreemptionPlugin simplified pseudo code}
calculateIdealAssignedResourcePerApp:
  for each app {
    app.idealAssigned = min((app.totalUsed + app.pending), queueTotalUnassigned)
  }

...

calculateToBePreemptedResourcePerApp:
  for each app {
    app.toBePreempted = app.totalUsed - app.idealAssigned - app.AMUsed
  }
{code}

For this use case, let's consider that all containers, including the AMs, will be 512 MB.
{{app1}} is running, consuming all resources in the queue. The RM creates {{app2}} and sets
its pending value to be 512 MB (for the AM).

Intra-queue preemption does the following during each round:
||AppID||priority||totalUsed MB||AMUsed MB||pending MB||queueTotalUnassigned MB||
|2|2|0|0|512|12288|
{noformat:title=FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp}
app2.idealAssigned = min((app2.totalUsed + app2.pending), queueTotalUnassigned)
                   = min((0 + 512), 12288)
                   = 512
queueTotalUnassigned = queueTotalUnassigned - app2.idealAssigned
                     = 12288 - 512
                     = 11776
{noformat}
||AppID||priority||totalUsed MB||AMUsed MB||pending MB||queueTotalUnassigned MB||
|1|1|12288|512|3584|11776|
{noformat:title=FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp}
app1.idealAssigned = min((app1.totalUsed + app1.pending), queueTotalUnassigned)
                   = min((12288 + 3584), 11776)
                   = 11776
queueTotalUnassigned = queueTotalUnassigned - app2.idealAssigned
                     = 11776 - 11776
                     = 0
{noformat}
{noformat:title=FifoIntraQueuePreemptionPlugin#calculateToBePreemptedResourcePerApp}
app1.toBePreempted = app1.totalUsed - app1.idealAssigned - app1.AMUsed
                   = 12288 - 11776 - 512
                   = 0
{noformat}

In this case, {{app1.toBePreempted}} should be 512 instead of 0. Since it remains 0, no container
ever gets preempted and {{app2}} can never get it's AM container.

I'm not yet ready to suggest a solution, but I wanted to point this out while we work on it.

> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>
>                 Key: YARN-2009
>                 URL: https://issues.apache.org/jira/browse/YARN-2009
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Devaraj K
>            Assignee: Sunil G
>         Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, YARN-2009.0003.patch,
YARN-2009.0004.patch, YARN-2009.0005.patch, YARN-2009.0006.patch, YARN-2009.0007.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need to consider
preempting the low priority application containers first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message