Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-dev@hadoop.apache.org
Date: Mon, 6 Apr 2015 20:43:12 +0000 (UTC)
From: "Ashwin Shankar (JIRA)" <jira@apache.org>
To: yarn-dev@hadoop.apache.org
Message-ID: <JIRA.12818693.1428352984000.7165.1428352992610@Atlassian.JIRA>
In-Reply-To: <JIRA.12818693.1428352984000@Atlassian.JIRA>
References: <JIRA.12818693.1428352984000@Atlassian.JIRA>
 <JIRA.12818693.1428352984302@arcas>
Subject: [jira] [Created] (YARN-3453) Fair Scheduler : Parts of preemption
 logic uses DefaultResourceCalculator even in DRF mode causing thrashing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ashwin Shankar created YARN-3453:
------------------------------------

             Summary: Fair Scheduler : Parts of preemption logic uses Defau=
ltResourceCalculator even in DRF mode causing thrashing
                 Key: YARN-3453
                 URL: https://issues.apache.org/jira/browse/YARN-3453
             Project: Hadoop YARN
          Issue Type: Bug
          Components: fairscheduler
    Affects Versions: 2.6.0
            Reporter: Ashwin Shankar


There are two places in preemption code flow where DefaultResourceCalculato=
r is used, even in DRF mode.
Which basically results in more resources getting preempted than needed, an=
d those extra preempted containers aren=E2=80=99t even getting to the =E2=
=80=9Cstarved=E2=80=9D queue since scheduling logic is based on DRF's Calcu=
lator.

Following are the two places :
1. {code:title=3DFSLeafQueue.java|borderStyle=3Dsolid}
private boolean isStarved(Resource share)
{code}
A queue shouldn=E2=80=99t be marked as =E2=80=9Cstarved=E2=80=9D if the dom=
inant resource usage
is >=3D  fair/minshare.

2. {code:title=3DFairScheduler.java|borderStyle=3Dsolid}
protected Resource resToPreempt(FSLeafQueue sched, long curTime)
{code}
--------------------------------------------------------------

One more thing that I believe needs to change in DRF mode is : during a pre=
emption round,if preempting a few containers results in satisfying needs of=
 a resource type, then we should exit that preemption round, since the cont=
ainers that we just preempted should bring the dominant resource usage to m=
in/fair share.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)