hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Jie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4462) FairScheduler: Disallow preemption from a queue
Date Thu, 14 Jan 2016 09:27:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097879#comment-15097879

Tao Jie commented on YARN-4462:

[~kasha] , Thank you for your comments. I will make some improvements as you suggested.
I have more consideration about:
Also, it would be nice to have a global option, so folks could pick a default enable/disable
based on the number of queues they have.
I am not sure if it is necessary to have a global option for enable/disable preemption from
one queue, since I don't think it is a reasonable scenario to set this default value to non-preemptable.
Now option for allowPreemptionFrom is set to true as default. In most scenario, we have most
queues preemptable and few queues non-preemptable, we just add option  allowPreemptionFrom
with value of false for those non-preemptable.
Suppose there is another scenario in which we set global option for allowPreemptionFrom to
true. We should have most queues non-preemptable and only few(maybe none) queues preemptable.
There are very large probability of failure to find candidate containers to be preempted.
And it breaks fairness among queues.
Also it would become confusing when we config queue hierachy. Now we set non-preemptable option
either in parent queue or in its leaf queue, the leaf queue is non-preemptable. We try to
find preemptable queues from root to leaf recursively and return when we get non-preemptable.
It makes sence. However, when we set global default value to non-preemptable and set one parent
queue or leaf queue value to preemptable, I would be confusing about whether the leaf queue
should be preemptable or not.

> FairScheduler: Disallow preemption from a queue
> -----------------------------------------------
>                 Key: YARN-4462
>                 URL: https://issues.apache.org/jira/browse/YARN-4462
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: Tao Jie
>            Assignee: Tao Jie
>         Attachments: YARN-4462.001.patch, YARN-4462.002.patch, YARN-4462.003.patch
> When scheduler preemption is enabled, applications could be preempted if they obtain
resource over they should take. 
> When a mapreduce application is preempted some resource, it just runs slower. However,
when the preempted application is a long-run service, such as tomcat running in slider, the
service would fail.
> So we should have a flag for application to indicate the scheduler that those application
should not be preempted. 

This message was sent by Atlassian JIRA

View raw message