hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4822) Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates
Date Thu, 24 Mar 2016 00:07:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209469#comment-15209469

Wangda Tan commented on YARN-4822:

[~eepayne], [~sunilg], [~jianhe],

Appreciate if you could take a look at latest patch, it contains a couple of refactorings:
- PCPP becomes 2 parts:
1) Basic code such as clone queues, record what to preempt and send kill event when max-wait
2) Candidates-selection policy, includes calculate ideal allocation and select preemption
- Original calculate ideal allocation and select preemption candidates goes to two classes
1) FifoPreemptableAmountCalculator is for ideal allocation calculation
2) FifoCandidatesSelectionPolicy is for how to select containers
- CandidatesSelectionPolicy and calculator needs to read some fields from PCPP, so I add an
interface for them to use, which is implemented by PCPP: CapacitySchedulerPreemptionContext
- Moved all configurations keys from PCPP to CapacitySchedulerConfiguration, so admin can
set configurations in either yarn-site.xml or capacity-scheduler.xml. (Ideally should be set
in capacity-scheduler.xml, however, existing user sets configs in yarn-site.xml. Since CapacitySchedulerConfiguration
reads yarn-site.xml as well, it is backward compatible change.)


> Refactor existing Preemption Policy of CS for easier adding new approach to select preemption
> --------------------------------------------------------------------------------------------------------
>                 Key: YARN-4822
>                 URL: https://issues.apache.org/jira/browse/YARN-4822
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4822.1.patch, YARN-4822.2.patch, YARN-4822.3.patch, YARN-4822.4.patch
> Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to select candidates
to be preempted (based on FIFO order of applications/containers). It's not a simple to add
new candidate-selection logics, such as preemption for large container, intra-queeu fairness/policy,
> In this JIRA, I propose to do following changes:
> 1) Cleanup code bases, consolidate current logic into 3 stages:
> - Compute ideal sharing of queues
> - Select to-be-preempt candidates
> - Send preemption/kill events to scheduler
> 2) Add a new interface: {{PreemptionCandidatesSelectionPolicy}} for above "select to-be-preempt
candidates" part. Move existing how to select candidates logics to {{FifoPreemptionCandidatesSelectionPolicy}}.

> 3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a chain. Preceding
PreemptionCandidatesSelectionPolicy has higher priority to select candidates, and later PreemptionCandidatesSelectionPolicy
can make decisions according to already selected candidates and pre-computed queue ideal shares
of resources.

This message was sent by Atlassian JIRA

View raw message