Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Wed, 16 Dec 2015 13:24:46 +0000 (UTC)
From: "Tao Jie (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12910603.1446694161000.38809.1450272286677@Atlassian.JIRA>
In-Reply-To: <JIRA.12910603.1446694161000@Atlassian.JIRA>
References: <JIRA.12910603.1446694161000@Atlassian.JIRA>
 <JIRA.12910603.1446694161856@arcas>
Subject: [jira] [Commented] (YARN-4333) Fair scheduler should support
 preemption within queue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059968#comment-15059968 ] 

Tao Jie commented on YARN-4333:
-------------------------------

In attached patch, we append app-level check for starvation stats to queue-level check in fairscheduler update thread. Only starvation due to fairShare is checked for app, since we don't have minShare for app. Also we record lastTimeAtFairShareThreshold for each appAttempt.
Now once one app is starving but the leaf queue is satisfied, it could also trigger preemption. 

> Fair scheduler should support preemption within queue
> -----------------------------------------------------
>
>                 Key: YARN-4333
>                 URL: https://issues.apache.org/jira/browse/YARN-4333
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: Tao Jie
>         Attachments: YARN-4333.001.patch
>
>
> Now each app in fair scheduler is allocated its fairshare, however  fairshare resource is not ensured even if fairSharePreemption is enabled.
> Consider: 
> 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource of queueA.  
> 2, Then the cluster becomes busy, but app1 does not release any resource, queueA resource usage is over its fairshare
> 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has its own fairshare, but could not obtain any resource, since queueA is still over its fairshare and resource will not assign to queueA anymore. Also, preemption is not triggered in this case.
> So we should allow preemption within queue, when app is starved for fairshare.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)