hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Yan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2395) FairScheduler: Preemption timeout should be configurable per queue
Date Fri, 29 Aug 2014 01:01:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114688#comment-14114688

Wei Yan commented on YARN-2395:

bq. Do we want to introduce backward incompatible change by removing the existing global "fairSharePreemptionTimeout"
config ?
Backward incompatible happens. Here the problem is, if we introduce per-queue preemption timeout,
the old "fairSharePreemptionTimeout" always be incompatible. To solve this problem, we can
check whether the old "fairSharePreemptionTimeout" exist in the alloc file; if existing and
without "defaultFairSharePreemptionTimeout" config, we can treat the "fairSharePreemptionTimeout"
as the default one.

bq. Just curious as to why we decided to inherit timeouts ?
Consider a scenario where we have one high priority parent queue and low priority parent queue,
and each parent queue will create multiple user queues runtime. The high priority queues want
to take the resource back as soon as possible, using a small preemption timeout; while the
low priority queues are ok for a larger timeout. This cannot be achieved if we only have one
default preemption timeout config.

> FairScheduler: Preemption timeout should be configurable per queue
> ------------------------------------------------------------------
>                 Key: YARN-2395
>                 URL: https://issues.apache.org/jira/browse/YARN-2395
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>            Reporter: Ashwin Shankar
>            Assignee: Wei Yan
>         Attachments: YARN-2395-1.patch, YARN-2395-2.patch, YARN-2395-3.patch, YARN-2395-3.patch
> Currently in fair scheduler, the preemption logic considers fair share starvation only
at leaf queue level. This jira is created to implement it at the parent queue as well.
> It involves :
> 1. Making "check for fair share starvation" and "amount of resource to preempt"  recursive
such that they traverse the queue hierarchy from root to leaf.
> 2. Currently fairSharePreemptionTimeout is a global config. We could make it configurable
on a per queue basis,so that we can specify different timeouts for parent queues.

This message was sent by Atlassian JIRA

View raw message