hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Thu, 10 Dec 2015 17:47:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051332#comment-15051332

Sunil G commented on YARN-4108:

Hi [~leftnoteasy]
Thank you for sharing the detailed doc and patch. Its a wonderful effort and came out nicely.
I went through the doc mainly, and checked few part of the patch. I will got through patch
in detail soon and share more comments if I have. 

Few major doubts:
1. With different {{PreemptionType}}, are we planning to handle preemption across queue, within
queue (fifo/priority), within user etc? YARN-2009 was trying to handling the preemption within
a queue adhering to priority. 
2. Currently all containers from a node is selected and tried to find which all are matching
the preemption type. Later {{selectContainersToPreempt}} helps to clear out the non-valid
>> I would see it will be a great help if flexibility is provided with some interface
to sort containers eventhough a great deal of validation is done. 
   Sorting parameter can be
    - submitted time 
    - priority of app (since we take bunch of containers from a node first, only few apps
in cluster will come in one shot)
    - priority of containers
    - time remaining for the container to finish (% of completion)

With these flexibility, user can tune which containers will be his first choice for preemption
provided all the size/user limit/locality are matched.
3. Could we get a choice to kill container based on data locality, I could see the changes
but cudnt see how its achieved in preemption manager end.

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4108-design-doc-v1.pdf, YARN-4108.poc.1.patch
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only
want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1
and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton
preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

View raw message