hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3510) Create an extension of ProportionalCapacityPreemptionPolicy which preempts a number of containers from each application in a way which respects fairness
Date Wed, 03 Jun 2015 00:14:49 GMT

    [ https://issues.apache.org/jira/browse/YARN-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570039#comment-14570039
] 

Craig Welch commented on YARN-3510:
-----------------------------------

[~leftnoteasy] and I had some offline discussion.  The patch currently here is simply meant
to keep from unbalancing whatever allocation process is active by, generally, keeping relative
usage between applications the same.  It doesn't attempt to actively re-allocate in a way
which achieves the overall allocation policy, i.e., "as if all the applications had started
at once".  (this is a more complex proposition, obviously).  There's a desire to have this
because, among other things, sometime down the road we may do preemption just among users/applications
in a queue and it will be necessary for the preemption to actively work toward the allocation
goals to do that, rather than just maintain current levels.  This will add some medium level
complexity to the current patch, deltas with the current approach are:
Since the effect of preemption on order for fairness doesn't occur until the container is
released, and we want to consider it right away, there will need to be a need to retain info
about "pending preemption" for comparison on the app resources (it will be a deduction from
usage for ordering purposes, as if the preemption had already happened)
The preemptEvenly loop will need to reorder the app which was preempted after each preemption
and then restart the iteration over apps (not necessarily over all apps, again, just until
the first preemption)


> Create an extension of ProportionalCapacityPreemptionPolicy which preempts a number of
containers from each application in a way which respects fairness
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3510
>                 URL: https://issues.apache.org/jira/browse/YARN-3510
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>         Attachments: YARN-3510.2.patch, YARN-3510.3.patch, YARN-3510.5.patch, YARN-3510.6.patch
>
>
> The ProportionalCapacityPreemptionPolicy preempts as many containers from applications
as it can during it's preemption run.  For fifo this makes sense, as it is prempting in reverse
order & therefore maintaining the primacy of the "oldest".  For fair ordering this does
not have the desired effect - instead, it should preempt a number of containers from each
application which maintains a fair balance /close to a fair balance between them



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message