hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4822) Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates
Date Mon, 28 Mar 2016 16:29:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214420#comment-15214420

Wangda Tan commented on YARN-4822:

Hi [~sunilg],

bq. I am not much getting much idea abt the use of having multiple policies. Bcz if one selected
candidate from policy1 can be vetoed by another policy (now we have only one), will it be
tough to explain. Here multi layer filtering is what happening, will it be fine we can use
one policy or a group of selected policy rather than looping all. Since we do not have a usecase
now, i think its ok. But it will be great if you could share some info on same.

I was planning to add a new selection-policy for reserved containers in YARN-4390. When multiple
policies configured in the chain, rear policy can only add new candidates instead modify already
selected candidates from front policies. You can take a look at WIP patch of YARN-4390 if
you have interests.

The reason of doing this is, we need to select preemption candidates to do surgical preemption
in some cases, so some policies should take precedence and policy with lower priority should
respect already selected candidates.

And such behavior is internal protocol of implemented protocols, I don't want to make it to
be pluggable in short term.

bq. From premption round to round. So do we need to keep this object per PreemptionCandidatesSelector
object. May be a generic one (static util class) is enough for whole PCPP?
I considered of this, but I think make it to be non-static could be more flexible, since we
could have sharable fields during calculation.

> Refactor existing Preemption Policy of CS for easier adding new approach to select preemption
> --------------------------------------------------------------------------------------------------------
>                 Key: YARN-4822
>                 URL: https://issues.apache.org/jira/browse/YARN-4822
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4822.1.patch, YARN-4822.2.patch, YARN-4822.3.patch, YARN-4822.4.patch,
> Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to select candidates
to be preempted (based on FIFO order of applications/containers). It's not a simple to add
new candidate-selection logics, such as preemption for large container, intra-queeu fairness/policy,
> In this JIRA, I propose to do following changes:
> 1) Cleanup code bases, consolidate current logic into 3 stages:
> - Compute ideal sharing of queues
> - Select to-be-preempt candidates
> - Send preemption/kill events to scheduler
> 2) Add a new interface: {{PreemptionCandidatesSelectionPolicy}} for above "select to-be-preempt
candidates" part. Move existing how to select candidates logics to {{FifoPreemptionCandidatesSelectionPolicy}}.

> 3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a chain. Preceding
PreemptionCandidatesSelectionPolicy has higher priority to select candidates, and later PreemptionCandidatesSelectionPolicy
can make decisions according to already selected candidates and pre-computed queue ideal shares
of resources.

This message was sent by Atlassian JIRA

View raw message