Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 17 Jan 2017 01:08:27 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13019711.1478740281000.20609.1484615307988@Atlassian.JIRA>
In-Reply-To: <JIRA.13019711.1478740281000@Atlassian.JIRA>
References: <JIRA.13019711.1478740281000@Atlassian.JIRA> <JIRA.13019711.1478740281507@jira-lw-us.apache.org>
Subject: [jira] [Updated] (YARN-5864) YARN Capacity Scheduler - Queue
 Priorities
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 17 Jan 2017 01:08:36 -0000


     [ https://issues.apache.org/jira/browse/YARN-5864?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wangda Tan updated YARN-5864:
-----------------------------
    Attachment: YARN-5864.branch-2.007_2.patch

> YARN Capacity Scheduler - Queue Priorities
> ------------------------------------------
>
>                 Key: YARN-5864
>                 URL: https://issues.apache.org/jira/browse/YARN-5864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-5864.001.patch, YARN-5864.002.patch, YARN-5864.=
003.patch, YARN-5864.004.patch, YARN-5864.005.patch, YARN-5864.006.patch, Y=
ARN-5864.007.patch, YARN-5864.branch-2.007_2.patch, YARN-5864.branch-2.007.=
patch, YARN-5864.poc-0.patch, YARN-5864-preemption-performance-report.pdf, =
YARN-5864-usage-doc.html, YARN-CapacityScheduler-Queue-Priorities-design-v1=
.pdf
>
>
> Currently, Capacity Scheduler at every parent-queue level uses relative u=
sed-capacities of the chil-queues to decide which queue can get next availa=
ble resource first.
> For example,
> - Q1 & Q2 are child queues under queueA
> - Q1 has 20% of configured capacity, 5% of used-capacity and
> - Q2 has 80% of configured capacity, 8% of used-capacity.
> In the situation, the relative used-capacities are calculated as below
> - Relative used-capacity of Q1 is 5/20 =3D 0.25
> - Relative used-capacity of Q2 is 8/80 =3D 0.10
> In the above example, per today=E2=80=99s Capacity Scheduler=E2=80=99s al=
gorithm, Q2 is selected by the scheduler first to receive next available re=
source.
> Simply ordering queues according to relative used-capacities sometimes ca=
uses a few troubles because scarce resources could be assigned to less-impo=
rtant apps first.
> # Latency sensitivity: This can be a problem with latency sensitive appli=
cations where waiting till the =E2=80=98other=E2=80=99 queue gets full is n=
ot going to cut it. The delay in scheduling directly reflects in the respon=
se times of these applications.
> # Resource fragmentation for large-container apps: Today=E2=80=99s algori=
thm also causes issues with applications that need very large containers. I=
t is possible that existing queues are all within their resource guarantees=
 but their current allocation distribution on each node may be such that an=
 application which needs large container simply cannot fit on those nodes.
> Services:
> # The above problem (2) gets worse with long running applications. With s=
hort running apps, previous containers may eventually finish and make enoug=
h space for the apps with large containers. But with long running services =
in the cluster, the large containers=E2=80=99 application may never get res=
ources on any nodes even if its demands are not yet met.
> # Long running services are sometimes more picky w.r.t placement than nor=
mal batch apps. For example, for a long running service in a separate queue=
 (say queue=3Dservice), during peak hours it may want to launch instances o=
n 50% of the cluster nodes. On each node, it may want to launch a large con=
tainer, say 200G memory per container.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org